01/20/2004: Emergency Maintanence - Router

[gavrielh]

We have some clients that are experiencing an outage of their websites running on NT26 and NT27. The outage has lasted over 4 Hours and we are experiencing downtime on our main website. Just signed up a large account today for 10 domains who cancelled their agreement with us based on our inability to get them setup this afternoon. This cost us many hundreds of dollars and future revenue in more than just the hosting arena. Why are some of these sites working and others not, when will this issue be resolved, what do we tell our clients, will we be issued a credit for this downtime since the total downtime exceeds the 99.9% guarantee? What will be done to protect the systems from this happening again in the future, shouldn't their be some redundant fail over system in place to protect our clients and our businesses?

[webeffx]

All of my sites just came back this is after noticing that they were down around 4 oclock central time.

I like Matrix and I am about to launch a new advertising campaign to promote my web hosting side of the company. All I need is to know Matrix has eliminated any single points of failure.

I understand network outages will occur but I need to be assured that Matrix is working to eliminate any single point of failure.

on a good note though keep up the Great Job!

[thecomputerpro]

Well, I hope all is well now that i can get to this post..

I have 2 comments for now and I am done.

1. Unexpected downtime is a fact of life and there is nothing you can do to prepare for it! Especially when it is hardware we are talking about.

2. As soon as I noticed this problem problem, I called tech support KNOWING they would be working on it! So it turns out that they were.

I have been here now for about 10 months and would not be anywhere else. This is this most downtime I have seen in one period of time while here!

Yes, reduntant system or replacement may have minimized the time but still we would have seen downtime. I apprecite the work matrix does to make my work easier.

We work in a tough field..... Nobody says thank you when everything is going good but eveyone yells when something goes bad.....

Moral of the story, Matrix still has my vote!

TCP

[Brangwyn]

http://matrixwebhosting.net/sla/

Credit/Downtime policy is explained there gavrielh

[admin]

Ok every thing is back up on a juniper and every thing is good to go. We are setting up a set of OpenBSD boxes for our new router / firewall as I am just sick of cisco.

Our cisco 7120 had a nic go bad just out of no where today so we are fully replacing so this should not happen again at least untill new hardware fails. But the core router will now be a xeon system so its about 100x the system a cisco system is. This also gives us a gige link for inside the network as well.

We will also be giving 15 day credits for any downtime today. Just send in a ticket to billing@vortechhosting.com and they will be happy to take care of that for you..

[Brangwyn]

I'm not complaining, but just wondering how you came up with 15days credits Brad ? The maximum credit days is 7 as I interpret the SLA becuase in excess of 8 hours (1 hour being 1 days credit) it should be a complete months refund shouldn't it ?

So working backwards from that, a 15 day credit would imply a 15 hour outage (in excess of the maximum 8 hours before the full months credit kicks in) doesn't quite tally up.

Again no complaints I know you guys have had a pretty bad day today, just trying to understand the logic.

[mdwatkin]

Like logic404, I also went through a few other service providers. I haven't found a provider yet that is completely immune to downtime except for servicers that host sites like Amazon, etc. But Matrix/Vortech has done a great job of keeping me informed, taking responsibility for its services, and working through the issues.

Though the communication misses every now and then (ie. timeout change w/o notice) I am extremely pleased with the value I am getting. Thanks Vortech!

[generic]

Just signed up a large account today for 10 domains who cancelled their agreement with us based on our inability to get them setup this afternoon.

Since you lost them why don't you forward the name so we can call on them.

I will offer you 25% of the first years hosting profits as a referral

[admin]

Originally Posted by Brangwyn

I'm not complaining, but just wondering how you came up with 15days credits Brad ? The maximum credit days is 7 as I interpret the SLA becuase in excess of 8 hours (1 hour being 1 days credit) it should be a complete months refund shouldn't it ?

So working backwards from that, a 15 day credit would imply a 15 hour outage (in excess of the maximum 8 hours before the full months credit kicks in) doesn't quite tally up.

Again no complaints I know you guys have had a pretty bad day today, just trying to understand the logic.

It has not been down for 8 hours stright so just to be nice I just went ahead and oked 15 days. Thing to remember is it must be 8 hours stright. Not 1 hour down, up 4 hours, down 4 hours, up 2 hours, down 6 hours. That would be a total of 11 hours but not stight. I thought that was clear in the SLA maybe not. But in any case it has not been down 8 hours ether. I think it went down around 4:30PM and on and off up till around 8:30PM or 9:30PM can't remember the time off the top of my head. But that would be 4 hours or so. So I am really giving a little more then I should to try and make up or help in this case..

[QuickStart]

Hi. I signed on with Matrix several weeks ago, but did not move any of my client sites. My former host was very reliable but when they did have trouble - there was nobody answering phones, emails or tickets. Today, I finally moved all my accounts over to Matrix. When I received notification of the main router failure, I was sick (first words, ah sh**!.

However, I logged into the forum and found that the issue was being worked on and I had specific information to provide my customers (who through some miracle did not call me) - where are the planets aligned today .

O.k. so here we all are. We all understand technology and know first hand the pain the rewards. The only way for resellers to survive is to have a provider that is aware, knowledgeable (sic) and willing to get the job done by communicating and fixing the problem. Thank you for the techs over there that are working the issue. If I could buy you a beer tonight - I would.

Go Pats!
Sam

[thecomputerpro]

Still have my vote....

And will not be asking for a credit!

TCP

[admin]

Originally Posted by webeffx

All of my sites just came back this is after noticing that they were down around 4 oclock central time.

I like Matrix and I am about to launch a new advertising campaign to promote my web hosting side of the company. All I need is to know Matrix has eliminated any single points of failure.

I understand network outages will occur but I need to be assured that Matrix is working to eliminate any single point of failure.

on a good note though keep up the Great Job!

Its really hard to 100% eliminate any single points of failure, but here is what we are doing a little different then before. Before we had our "main" router a cisco 7120 that did our routering just for inside the network, our core switch is a cisco 3500 this can also do routing and "should" of been able to be the back up to the router. We never tested this live because it would of taken stuff down. This was what cisco recomend to us as a good backup option if the router had an issue we could just switch over to the switch/router.


So now for what we are doing. We are setting up a openBSD router/firewall we are also building a 2nd box just as the first one that will also be a openBSD box with the same setting config and all. If the first one does have a hardware issue we will be able to switch over to the other box with little down time ( nothing like today ) and be back up and running. But there is no easy way to run 2 routers or 2 firewalls at the same time. It can be done but at some point you will end up with a single point any way. But we are trying very hard to EVER keep that from happening like today.


Just like any one else we do try to plan things the best we can but some times things just don't seem to work like they were said to or should have. Also remember that we have grow a "LOT" since this router and switch setup were done. Now we are making sure that the first one can take the load and are putting the same thing right under it the rack. So there is no question as to if it can do it or not, maybe a little better planning on our part this time. Live and learn I guess and make sure it does not happen again.. Thats what i am doing now..

[admin]

Originally Posted by QuickStart

If I could buy you a beer tonight - I would.

Well I don't drink beer but a Dr. Peper would be good..

[zennix]

Hi, while I really like OpenBSD you are not going to get much use from a dual Xeon with it. OpenBSD does not support dual processor systems. It will work, but the extra cpu is going to be wasted. You can check their support page here:

http://www.openbsd.org/i386.html

[joshcombs]

I am satisfied with Vortech's response They have great customer service