Router / Network Down Time On 9/10/2005 between 1:00am ET and 2:00am ET

[MEELAN]

all seems normal for me here except the " vortechhosting.com [216.81.79.9]" entry in tracert. earlier it was showing "rapidcolo.com". I think we all preferred to have "rapidcolo.com" instead of "vortechhosting.com [216.81.79.9]" in tracert and IP lookup on earlier discussiom.

look faster too but it is saturday evening in this part of the world ;-)

[powvex]

a reply from admin is appreciated, is this done?, are we good?

[Vantage]

We are all done.

A few of our BGP sessions arent up yet, But we should be 100% reachable now.

P.S. The Bump Earlier was just to get this back on the top 10 so people would remember what was going on.

[powvex]

we are getting 50% packet lost. (or more)

[powvex]

Also round trip time has got upper than before; it was about 25ms from EV1, and now its around 35ms.

Round trip time to 216.157.145.2: 34 ms
Round trip time to 216.157.145.2: 35 ms
Round trip time to 216.157.145.2: 34 ms
Round trip time to 216.157.145.2: 35 ms
Round trip time to 216.157.145.2: 34 ms
Round trip time to 216.157.145.2: 35 ms
Round trip time to 216.157.145.2: 35 ms
Round trip time to 216.157.145.2: 35 ms
Round trip time to 216.157.145.2: 35 ms
Round trip time to 216.157.145.2: 35 ms

[powvex]

Maybe this help, some timeouts after epik network:


......
9 27 26 27 216.22.67.193 miamfllr2m6-so-1-0-0.ip.epik.net
10 31 31 31 216.22.67.221 tamsflde2m6-so-3-0-0.ip.epik.net
11 33 33 33 216.22.67.213 orleflcb1m6-so-3-0-0.ip.epik.net
12 33 33 34 216.22.67.165 orleflcb1m2-ge-0-0-0-117.ip.epik.net
13 Timed out 33 33 216.22.65.102 -
14 34 Timed out 34 216.81.79.9 rapidcolo.com
15 35 34 35 216.157.145.2 cp.hsphere.cc

[powvex]

many servers are down, is this related too??

[Romme]

I'm also having about 25-50% packagedrop

Here's the last part of my traceroute
11 115 ms 115 ms 115 ms sl-bb21-atl-6-0.sprintlink.net [144.232.20.176]
12 116 ms 116 ms 116 ms 144.232.29.66
13 123 ms 123 ms 123 ms leg-208-30-205-130-RIA.sprinthome.com [208.30.20 5.130]
14 123 ms 123 ms 123 ms border8.ge2-0-bbnet1.acs.pnap.net [64.94.0.16]
15 * 134 ms 134 ms vortech-1.border8.acs.pnap.net [216.52.194.78]
16 160 ms 140 ms 138 ms vortechhosting.com [216.81.79.9]
17 135 ms 136 ms 136 ms hsphere.cc [216.81.70.2]

[powvex]

nt8, nt18, nt17, nt22, nt31, mssql8, ... Are down. Whats wrong??

[kattouf]

Yupp Nagios is showing 8 servers down..

[obs]

Any update? Some of our sites are still down.

[powvex]

its been 4 hours and almost all sites have problems, and servers are down.

whats wrong? could anybody tell us please?

[Vantage]

Hello,
If you have a site down, Submit a ticket.

Everythign appears to be working at the moment.
Mail7 and Mail8 were showing down because they were accepting there maximum number off connections for a while, getting all the mail that was waiting fromthe network outage.

I have restarted all the NT servers that were down.

[Selrahc]

In same boat as everyone else. Once inside RapidColo.demarc.cogentco.com packet loss ranges from 33 to 50% on the indise hops. Thats over 100 tests using pathping. However I'm sure they are working hard to correct the problem. Im going to get some sleep and hope everything is ok by morning

[powvex]

Vantage, I know the servers that were down are up now, but we need explanation for 4 hours down on almost 10 servers, not to tell us they are "working at the moment", as maximum expected 1 hour on downtime.

also we have great packet loss at the moment.