Network Outage 6/8/07

[bubba]

Nothing like a good old DoS attack to get the community back together. I haven’t been reading the forum much in the last few weeks and months because things have been very stable in the VT world. (thank you)

Good to see all the familiar names again.

I wish there was a switch to flip to go to a backup set of servers when these things happen. What do the larger providers do differently to avoid a complete blackout of their entire cluster?

Any ideas on preventive measures we can take as individuals would be interesting to read. In my case I would only need to assign this additional safety net to certain customers. Many of my clients are not heavy users of their website and email. Many of these users do not even know when their site is down.

It would be a waste of resources to provide redundancy for these users but some clients depend on their site to conduct the bulk of their daily business and offering a backup solution would be something these clients would gladly pay to have.

b

[promitech]

are we under another attack? one of the mysql servers seems to be down I think it is mysql9 or mysql8

Rick

[mister*spark]

having trouble with unix28 again.. but you probably knew that already.

good luck..

[promitech]

I am not liking this. I have 200 customers on my A$#$

[promitech]

Service Status Details For All Hosts


Host Sort by host name (ascending)Sort by host name (descending) Service Sort by service name (ascending)Sort by service name (descending) Status Sort by service status (ascending)Sort by service status (descending) Last Check Sort by last check time (ascending)Sort by last check time (descending) Duration Sort by state duration (ascending)Sort by state duration time (descending) Attempt Sort by current attempt (ascending)Sort by current attempt (descending) Status Information
4-3A-R5-S1


PING

CRITICAL 06-09-2007 00:02:12 0d 10h 53m 0s 1/4 FPING CRITICAL - 66.103.149.18 (loss=100% )
CF2


CPU Load

WARNING 06-09-2007 00:02:44 0d 0h 2m 27s 4/4 CPU Load 50% (10 min average) 91% (60 min average) 49% (1440 min average)
Colo28


PING

CRITICAL 06-09-2007 00:03:51 1d 11h 42m 42s 1/4 FPING CRITICAL - 216.157.142.224 (loss=100% )
MAIL11


SNMP Disk Check

CRITICAL 06-09-2007 00:03:49 0d 10h 4m 23s 4/4 /usr/local/vpopmail/domains: less than 10% free (= 90%)
MAIL2


SNMP Disk Check

CRITICAL 06-09-2007 00:04:50 0d 2h 48m 22s 4/4 /hsphere: less than 10% free (= 90%)
MSSQL


Check NT Disk C

WARNING 06-08-2007 23:56:09 10d 18h 53m 31s 4/4 C:\ - total: 17.09 Gb - used: 15.57 Gb (91%) - free 1.52 Gb (9%)
NT


Check NT Disk D

CRITICAL 06-09-2007 00:02:32 0d 13h 12m 43s 1/4 D:\ - total: 34.18 Gb - used: 32.75 Gb (96%) - free 1.43 Gb (4%)
NT17


Check NT Disk D

WARNING 06-08-2007 23:57:33 0d 12h 17m 40s 4/4 D:\ - total: 34.19 Gb - used: 30.92 Gb (90%) - free 3.27 Gb (10%)
NT32


Check NT Disk D

WARNING 06-09-2007 00:03:30 3d 18h 12m 11s 4/4 D:\ - total: 34.18 Gb - used: 30.96 Gb (91%) - free 3.22 Gb (9%)
NT35


Check NT Disk D

WARNING 06-09-2007 00:03:39 11d 2h 20m 25s 4/4 D:\ - total: 34.18 Gb - used: 32.04 Gb (94%) - free 2.14 Gb (6%)
NT44


Check NT Disk C

WARNING 06-09-2007 00:03:39 2d 5h 41m 34s 4/4 C:\ - total: 34.18 Gb - used: 31.04 Gb (91%) - free 3.14 Gb (9%)
NT51


CPU Load

CRITICAL 06-09-2007 00:02:33 0d 5h 2m 41s 4/4 CPU Load 100% (10 min average) 100% (60 min average) 61% (1440 min average)
NT6


CPU Load

CRITICAL 06-09-2007 00:02:00 0d 0h 38m 12s 4/4 CPU Load 99% (10 min average) 90% (60 min average) 69% (1440 min average)
NT7


CPU Load

CRITICAL 06-09-2007 00:01:11 0d 0h 39m 0s 4/4 CPU Load 99% (10 min average) 92% (60 min average) 66% (1440 min average)
NT9


Check NT Disk C

CRITICAL 06-09-2007 00:01:58 1d 5h 23m 13s 4/4 C:\ - total: 34.18 Gb - used: 33.27 Gb (97%) - free 0.91 Gb (3%)

[shadowfyre]

Are we down again?

[promitech]

looks like it to me. looking for somewhere to move to...

[mdallarosa]

This errors do not necessarily affects service availability. If you check in Nagios, all servers are up and running.

It seems some servers are under attack again so their ip`s has been disabled while they try to control it. At least unix27 an unix28 seems to have their ip´s disabled.

[promitech]

I see were not getting any feed back from vortech as to what is up

[mdallarosa]

unix27 is reachable now.

[mister*spark]

unix28 down again

[jours]

Guys, I don't want to be a back-seat driver here while you're trying to solve this (and I really appreciate the efforts, Vortech is head-and-shoulders above everyone else in support) but...

It's the third night we've seen unix28 go down now. Same time, presumably same attack. If someone actually intends to attack Vortech it seems they would target more than just one machine at this point, which leads me to believe that this is targeted at a particular site on that box.

In my experience with this type of thing the resolution has been to rotate the shared ip(s) on the box and then try to correlate the DNS logs with attacking addresses as the zombies update (looking for coincidence by ip, time, or even geographic anomaly if the zombies are still being actively controlled). Obviously if you can isolate the target site (or candidate sites) then you've got a bunch of options for restoring service to everyone else.

Take it FWIW...obviously I'm not in the fray there so I may be way off base here. I'd love to hear a recap of what happened once the dust settles.

Keep up the good fight!

John

[MartinGavin21]

Hey guys, I totally agree with jours.. Ive been with you guys for a bit of time now and the service has been great but over the last couple of weeks there have been quite a few problems and the base of these problems seem to be unix28. As it stands now I can not reach any of my sites on that server and one of them is due for presentation / launch this morning which is not going to look good for us if we have to tell the client that their site is already down before its live!!

Can we have an update on an estimated time this server will be back up and also if it is an option to have the server replaced?

Martin Gavin

[Danl]

Hey guys, I apologize for not putting this up on the bulletin, I wanted to make sure it got fixed as quickly as possible, and surprisingly enough everything was only down for about 20 minutes (except for unix28 which was down for an hour). I've been watching it really closely at night, and the best thing that I can personally do is make sure it stays up. We're looking into several different options for stopping this once and for all. I don't really know which option we're going to choose or even all the different options, but I assure you all that we will have these problems solved as soon as humanly possible.

-Dan

[TheNaj]

Any updates on this issue? I am still experiencing intermittent outages that are causing shopping cart errors and other problems.

I am having a hard time explaining these issues to my clients. Please Advise ASAP.