2004-09-19 Windows servers

Someone appears to be running a DoS attack against NT31 and is taking out the whole cabinet with it. We're working on it.

Actually, from a 3rd party observation, its taking the whole network Get em' Alex.

[bootNumlock]

I'm with Matt on this one... the graph's look a little orange and most of my sites are creepin'

[tkraffty]

ouch - seeing ups and downs across the board, customer already called about email...

guess you're rebooting routers/machines etc. or your DoS friend is being pretty damn systematic and/or hogging the whole pipe.. egad

He's being methodical and is rotating hosts/networks. James has spent the past hour shooting them down as they come in. We'll try to divert traffic away from that switch altogether and let the router take the load. Service to NT31 will be iffy at best for the time being.

[antic]

Hey guys, good luck! About how long does it take for a DoS attack to clear up? Or for whomever it is to get bored?

[Brangwyn]

Most DDoS's use "zombie" machines.. usually some twit just sets off the attack and letts the zombie machines do all the work, some can last days before they finally stop.

[cambodia]

i heard if we use best equipment with fast alert it never run longer than 15minute ?

Originally Posted by cambodia

i heard if we use best equipment with fast alert it never run longer than 15minute ?

Well, with unscrupulous invidividuals selling hordes of zombie machines on the internet, depending on their intent, they could keep even the most equipped networks down for hours on end. Eg. Akamai, SCO, etc.

20,000 Zombies - 2000$ ...

http://www.usatoday.com/tech/news/co...bieprice_x.htm

Of course, Alex and James seemed to have gripped it fairly quick as I didn't notice any prolonged outage from here... A few blips here and there for about 20 minutes, but I didn't see anything too terrible

[Vantage]

Everything should be working well now...

Im going to be monitoring it on and off all night.. Just to be safe.

p.s.
There is no way to stop a big enough DDoS. So far I have blocked over 5500 IPs and all UDP service to the box.... Looks to have mostly stopped it but..... They are still trying...

[tkraffty]

Hi - had a 1/2 hour outage reported from a site on NT18 this morning at 8:45AM. Is these issues still related to the DoS attack? Just wondering if there's still some lingering effects, and what my day is going to look like in terms of customer suport

[Silverbug]

So far I have blocked over 5500 IPs and all UDP service to the box....

so when this happens you just block the ip of the computer that hammering the server?

[Vantage]

Silverbug,
It depends on the specific incident.
In this case we had a wide variety of IPs all from the same geographic area. They were using a number of attack methods and there attack seamed to be localized on one IP (Shared IP of NT31).
Due to the type of attacks we were seeing it was difficult to block the attackers in "one fell swoop".
I dont want to bore you with the whole story but it appears that they were prepaired to take down that server by any means they had available.
The only real choice I had was to Block large blocks of IPs. After I had blocked about 5500 IPs they decided to go with a spoofed UDP flood. UDP is VERY easy to spoof and it doesnt realy matter where the traffic originated from.. if the IP they are spoofing isnt blocked then the attack will get through, even if you are blocking the true IP of the attacker.
This is when I rate limited the default NT31 IP and blocked all UDP... 20 Min or so later they gave up... They IPs remain blocked until I stop seeing nasty traffic comming from them...

[Brangwyn]

Theres not really a lot else ya can do SilverBug, though most routers do provide tools for mitigating attacks as well like SYN packet limiting etc also.

[Brangwyn]

Just a side thought here .. is there actually anything other than DNS running that would actually even require UDP to get in past the border router? could UDP not be completely dropped other than UDP53->DNS Servers?