2004 - 30 - 10 Router Outage

[Carly]

Originally Posted by TonyK

Ok can you please tell us what happen Carly?

I don't have the full details yet, all I know is that our main router failed and the Network Admin had to set us up on a temporary solution. They're currently working on getting the main router back online as soon as possible.

An admin will post later with full details. All sites should currently be up. If anyone is experiencing any problems, please submit a trouble ticket. There may be another brief period of downtime when we are able to switch back to the main router. We will be announcing this as we go.

We sincerely apologize for the inconvenience and assure you we're doing our best to get everything running as it should.

[party1957]

Originally Posted by Carly

An admin will post later with full details. All sites should currently be up. If anyone is experiencing any problems, please submit a trouble ticket.

Well every site of mine is still down...

Guess I'll go submit...

[Brangwyn]

What I suspect your seeing party1957 is just a result of the temporary solution not coping too well with the load, I'm getting the DNS servers timeout fairly frequently which would cause you to get a DNS style error in your browser whilst trying to view your sites even though they (the servers) and the network are physically up currently.

[tkraffty]

Looks back up, across the board - but (1.) cringing it took about 3 to 4 hours to begin resolving, and (2.) seems slow (probably and understandably due to the temp solution mentioned being in-place right now)...

Just one question: What about all that redundent/secondary router hardware you installed earlier this year? (Why didn't that help provide any immediate bipass to this primary router failure tonight?)

[admin]

I am not seeing anything else down. If something is still down please submit a ticket or post the sites here so we can look in to it.

[Brangwyn]

It's just rather slow all around Brad, and thats giving the appearance of things being down, i'm guessing some packet loss causing DNS queries to go unanswered as I mentioned above.

[Vantage]

THh issue was with our core router. We had some hardware failure and it locked down the OS. We have it solved now and are waiting to roll back over to the main unit till the begingin of the week.
(Giving the router some uptime to make sure the issue is resolved)

[craigdunlop]

Was there redundancy in place for the core router or is this not possible?

[Vantage]

There was redundancy in place... The secondary routing engine failed also...
Our third redundant option is being put in place... But part of it relies on internap... they have delayed the turnup of there connection.

[admin]

I have shut off IP check right now because of all of its SNMP stuff was killing stuff I have setup http://66.194.198.60:8080/ I think I have about 99% of the servers. If your site is or goes down please be sure to put in a ticket ASAP.

[Light Speed]

Right now pages are loading really slowly and it's 1am PST. Is it going to be like this for a few days? If so then what's it going to be like at peak times?

Yeah, everything is really slow right now:

Pinging www.matrixreseller.com [216.157.144.17] with 32 bytes of data:

Reply from 216.157.144.17: bytes=32 time=199ms TTL=49
Reply from 216.157.144.17: bytes=32 time=156ms TTL=49
Reply from 216.157.144.17: bytes=32 time=139ms TTL=49
Reply from 216.157.144.17: bytes=32 time=144ms TTL=49

Ping statistics for 216.157.144.17:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 139ms, Maximum = 199ms, Average = 159ms

Normal is about 40ms from my 6mbps cable connection in Seattle....

All sites seem to be up but are virtually unusable - on dial-up it's about 30 - 45 secs for a page to load (so that's 30-50% of customers gone) and about 10 secs on 1.5mbs cable from the UK - after the cabinet problem (I'm on NT4 unfortunately) last week, this is bad news - please don't delay the main router as for us e-commerce people, all the sites are now effectively down and will be until the problem is resolved

[jmh]

After the dust has settled, could someone please check into the "offsite monitor & forums"?

The biggest frustration was not knowing what was going on, and not being able to refer to the offsite tools that were established for just this sort of occasion. The offsite IP monitor would occasionally respond after 11 PM (showing everything down), but the forum wasn't accessable until the problem was solved.

If the "backup notification" system is indeed off-site, perhaps a better provider needs to be found. If it is on-site, it will likely suffer the same outage as the rest of the system, making it inaccessable (and useless) as well.

All in all, it was (and is) a great idea, but it didn't work last night when we needed it.

Thanks!

[Brangwyn]

The backup forum was available for the first hour or two of the outage, Carly mentioned earlier she wasn't sure what was up, but a reboot of that box seemed to bring it back to life. I suspect now the incidents been and gone that working out why it went down probably won't be all that easy.