11/14/03 Data Center link

There was a connectivity issue with our upstream provider today, this should now be resolved.

I couldn't post this until now as our link was down .

The link is back up, this post is to let you know what happened.

[mresell]

nice to know I wasn't crazy...after yesterday.

[treeves]

How many upstream providers does Vortech have? Shouldn't there be more than one for redundancy, as stated in Vortech's network explanation-

"Studies have shown that the most common reason for downtime is circuit failure on Tier 1 provider backbones, the major data highways. To guard against this potential problem, we have two Tier-1 providers. If one experiences problems, we can route traffic down another one. "

Sounds like we just had one of these problems.

[admin]

Seems there was an issue with the core juniper router. It should all be fixed now we also rebooted our cisco router to make sure there was no issues there...

[admin]

Originally Posted by treeves

How many upstream providers does Vortech have? Shouldn't there be more than one for redundancy, as stated in Vortech's network explanation-

"Studies have shown that the most common reason for downtime is circuit failure on Tier 1 provider backbones, the major data highways. To guard against this potential problem, we have two Tier-1 providers. If one experiences problems, we can route traffic down another one. "

Sounds like we just had one of these problems.

It was not the providers that went down that is what the qoute you have is talking about it was the core router that was having an issue..

Cogent, Level(3), Epik, and qwest were all still up.. We have fixed the issue and should be good now. This is the first unplanned outage in the past 6 to 12 months I can even think of..

[onsitetech]

Whatever else you did it's flying...sweet!

[zennix]

Any idea how long this was down? Btw, syntax such as "seems to be" and "should be" are not exactly comforting. Site is screaming fast though! I have to remind myself that my databases are on a seperate server the queries hit so fast, feels like they are running locally. Anyway, glad it is back up this is the first little blip I have seen with you guys.

[admin]

I say should be or seems to be because I was not 100% sure what happend in the first place when I posted..

But I have now and it seems there was something on the juniper router that caused the issue..

[mresell]

When you have that many servers to care for and you have an outage thinking that you are 100% sure at any given moment of what isn't responding properly would be a pretty impressive task. Networking is complex for a reason. Kinda like the human body the symptom may not be directly related to the cause..at first anyway. Hey you guys appear to be more informed about your network than tons of larger providers. Glad it's up!

[nelsonke]

Thanks for the clarification. My most influencial reference client is on NT26. We have had all the asp outages that everyone else has had, PLUS, two other outages due to actions taken by Matrix (which y'all have apologized for) relating to a specific dll we had you load for this client, and now this total outage. Wow, it has been tough. (The client called within minutes of going down)

While this has been difficult, your willingness to be "transparent" and provide the ipMonitor has made this manageable. When the client called I immediately went to the monitor and could tell them that the whole data center was down, and that I was confident that everyone was working on it.

My last hosting wholesaler had no such monitoring facility, nor this kind of forum. That is why I am here not there. My hat is off to you, keep up the good work guys.