01/20/2004: Emergency Maintanence - Router

Our main Cisco Router is experiencing unexpected hardware failures. This is causing intermittent downtime for all sites on our network. We are currently working on stabilizing the router, so that it will remain up for the next hour while we prepare a replacement.

Once the replacement is ready, the network will go down again for only a few minutes while we switch out the router. We apologize for the inconvenience.

[resell01]

So what was the total downtime: 4:55pm to 5:35pm ???

[Brangwyn]

I think it was a little longer than that, I first encoutered problems about 2 hours ago, most things appear to be working atm, but pretty slow response times.

[tkraffty]

When things settle down again, I'd be curious to hear whether or not it is possible and/or in the works to have an alternate router.

It seems we have a router failure roughly every couple/few months... and once that happens, every one of my customers are left wondering why they never saw this same sort of problem at the other web hosts I've talked them into leaving for my service.

I thought we had some sort of backup or redundency that would allow an alternate router to kick-in automatically - instantly - whenever something like this happened... but if not, this seems like a pretty significant vulnerability in the entire network -- not so much in terms of hardware configuration as much as this problem relates to a simple fact:

My clients tend to forget the months and months of great service provided here after about 10 minutes of finding their web site and e-mail unavailable.

I'd appreciate knowing whether or not some backup/redundent system is in the planning there at all...

Thanks

Hello Guys,
Time for an update.

We've taken the router out of the loop and have our network hooked up through our core switch instead. This has caused a significant load on the switch, so we are in the process of setting up a stateful packet filtering firewall on a dual 2.4ghz Xeon, running OpenBSD I believe. This will offset the switch's load and should return us to 100% operational state on the network.

Now, the core router our backbone provider (Level3) is going to be taken offline for about 30 minutes. They have given us some time to let you guys know of this issue. Apparently one of their UPS cabinets is having some major issues and must be resolved immediately. Not too sure on the details, but Brad might fill us in later.

So to sum it up, our network is up, and we'll be installing the Firewall later tonight, meanwhile, the core router for Level3 is going down for 30 minutes. :/

[resell01]

So another 30+ mins. of downtime...Exactly when?

[onsitetech]

Thanks!

Rick
OnSite

[Brangwyn]

Thanks for the update.

[electricfox]

Just wondering if this router downtime should be affecting all sites, or just some - one of my client's sites is down (freshbait.com) and I'm not sure if it is related or not. Don't want to bug you if it is.

[s58smith]

How is that the web server for this forum works so well? I can get to this forum right away and everything is very fast.

I guess I must not understand everything. I get some of my client's web sites to come up, and not others. (Around 5pm) Now different sites are working (slow), but the ones I orginally could see don't work. Seems traffic should effect all my requests the same.

Why is it that your site is not effected like web servers seem to be hosting our clients' sites? The connection to it seems redundant and working good, so should the connection to other servers be as good.

Thanks,
Steven

[logic404]

Suggestion...is it possible to have some kind of "opt-in" email list to get these kinds of things. I know you wouldn't want to force on everyone telling them about every little thing, but for those of us that want to know, it can be useful. Because during this latest outage, one of my clients rang me to ask what the problem was, and of course the forum was down as well. So I just had to spin something along the lines of "there's been a hardware failure and techs are working on it" - cause you guys are so good, I knew you would be!! But it would have been nice to know of the issue as it happened...just a thought.

But keep up the top notch service - while I was on the phone to the same client, the site came up. That made him pretty darn impressed!

We do have an announcements list:
http://vortechsupport.com/ click "announcements" and subscribe.

Not sure about the core router yet, will let you know if we know when/if it will go down.

The slow/intermittent website issues are due to the high load being placed on the core switch at the moment, which we are working to offload.

[toomanylooms]

I installed a new customer this morning. By 4pm this afternoon, the customer was flippin and said one of his requirements was a "reliable" website. So... bad timing. but if I had known something, I could have responded.
The messages say it will be 30min more at 4:35 (eastern time?) but its not 8pm eastern time and I still can't FTP. When are things going to be fixed.

[logic404]

Ask and ye shall receive! Thanks Matt - that's service - it was there and I didn't even know it!!

toomanylooms - I know this may not be a comfort to you now, but having been through a few hosts, and my oldest client having been through them with me, it is SO good to be able to say "the techs are working on it right now" and KNOW that they are. Even though I wasn't on this forum or anything, I could say with complete confidence that the guys at Matrix were doing everything they possibly could to fix the problem. I reminded my client of previous providers where I wasn't able to make that assurance and they were quite OK. Just a thought...(I'm Matrix happy right now - if I could afford it, I'd fly to Orlando and give you all a big hug!!)

[resell01]

Slow.....I have some sites not coming up at all....Unix4 for one.