2004 - 30 - 10 Router Outage

[Vixen]

Yes, the swap has already happened.

I missed Halloween because we had to go to Clearwater for parts and then I was at the office & data center until 5am this morning. <sigh>

[jmbeach]

I take it no kids trick-or-treat around the data center...

[Vixen]

We didn't get down there until after 2am so all the kiddies were passed out after their sugar buzz by then. LoL

[mresell]

Hey, I could of missed passing out candy. They wore me out. They were ringing the bell like crazy...couldn't get to the door fast enough. I actually hate candy and giving it out, but some years it gets hard to find alternatives like toys, tatoos etc... Americans can find more ways to poison themselves than anybody.

[Ram_Argid]

First of all: Carly, Thanks for the early status update

Originally Posted by Vixen

Francisco, everyone was at the data center working on the issue which was why you got voice mail. Generally, when there is someone *major* going on everyone is actively working on the issue so submitting a ticket and/or checking the forum is the best way to contact us during issues like this.

The trouble is, and I understand Francisco's frustration and point here, that one could not submit a trouble ticket, view the forum, or contact you in any way with the entire network down. This means that the phone was the only way. It would always be nice to have someone answer the phone and purr in your ear about how everything is going to be ok. However, that would take someone off the task and you still wouldn't be able to answer all the calls.

I'd rather see the problem being worked on. Although some way to at least get some feedback would be greatly appreciated.

Originally Posted by Nightfall

How are we compensated for all these outages which directly affect our sites, our clients' sites and our business reputation as a whole.

Nightfall - I have been around and have "experienced" enough hosting providers to tell you this: Vortech (and the team) bring a significant amount of value and support to the table, and they do it inexpensively. I understand your frustrations, however, should you be so interested in a totally infallible system (which is a myth by the way) may I suggest that you pay for it?

There are plenty of "premium" hosting companies out there that will gladly take 300+ of your hard earned dollars every month, providing fewer services, albeit guaranteeing up time.

The Vortech team works their collective butts off, and are constantly improving their services.

I too lost sales/income from the recent issues, and it hurts. However, Vortech is the BEST, most RESPONSIVE hosting provider I've come across, and furthermore, they don't rape you to provide this service. I for one will NOT be hounding them for compensation, I believe that they responded admirably to a bout of bad luck and the service is better because of it.

There should have been a redundant system in place, it too failed, it's technology...sometimes it fails. I'll bet this particular system doesn't fail again though, a body can only take so much pain....Isn't that right Vixen?

Originally Posted by Pace

Due to impending elections Vortech has been amBUSHed...:-)

Pace - C'mon, stay focused. Personally, I'd prefer to keep politics out of these issues. There are plenty of forums out there for your personal opinions.

Originally Posted by Vixen

I am not trying to be bitchy but I think it was best to get the network back up to speed and stable.

48+ hours...missing halloween - I think you're entitled to a little bitchyness
-----------------------------------------------------

TO THE VORTECH TEAM:

The outages hurt, but I thank you for all your hard work and improvements. It's dedication like this that is missing from many companies today. I for one remain a satisfied and loyal customer.

Remember people: No company is without problems and challenges; it's how the company deals with those problems and challenges that truly defines the company.

Keep up the good work, and get some sleep

[aaronray]

One final question related to last weekend's outage:

What's the status of improved router redundancy? I thought I read somewhere that an update would be forthcoming, but if it's here, I seem to have missed it. Any information (or a link to the info) would be appreciated. Thanks.

[Vantage]

Hello Again,
Sorry, I must have been slacking with the updates... :-)

Our core router has now had the failed hardware replaced. We have also stocked a completely seperate set of redundant hardware. So if both the primary AND secondary hardware should fail again we have a completly seperate set of primary and secondary hardware to replace it with. The second set has a mirror of the current config already on it. so it is "plug and play".

Basicaly we have 4 levels of redundancy now... (we have an entire rack worth of JUST routers now... )

[Light Speed]

Originally Posted by Vantage

Basicaly we have 4 levels of redundancy now... (we have an entire rack worth of JUST routers now... )

Sounds like a kodak moment

Any pics?

[Vantage]

Ill get some this week if possable. Right now the cabinet is NASTY looking. The nice pretty wireing job BladeSnitz had done is all in Shambles.. If he saw a picture he would kill me.. I am gonna clean it up first...

Its amazing how much wire a 48 port card can have in front of it...