So, what exactly happened?

[jmbeach]

What was it that caused that 20+ minute outtage? Looked like the whole operation went down simultaneously.

[xweb]

Well, as my client just said to me "You get what you pay for."

You think if I charge them more they would still be ok with the downtime?

I know I cannot complain too much at 35 smackers per month.

[nhdonny]

We just had another for about 5 minutes

[admin]

xweb, you can complain I don't mind.

But just so every one knows what is going on.

Around 3am a users of ours using SSH hacked a server and has tried to hack all unix systems in the h-sphere cluster. Because of this we have shut SSH off network wide to every one.
We will not longer be able to offer SSH access to any of our systems because the abuse this is the 3rd time and I would rather all users be safe from this.

SSH will be turned off for users on every system, and blocked at the router unless its a colo server it will be open to them. When we did the upgrade to the router something went wrong and locked every thing out when we blocked SSH, so we are having anther cisco tech take a look at. Every thing should be fixed right now and we are still trying to get SSH blocked at the router.

[jmbeach]

okay, thanks

[admin]

I hate having to disable ssh but its ether that or we all get hacked.. We are looking at setting up jails for SSH that may help but at this time it just has to be off.

[somereseller]

This is totally unacceptable to block ssh for everybody on all servers without notice. I depend on it and so do some of my customers. I know personally all of them and I'm the one using ssh on their account most of the time.
If it's for a couple of days, we may be able to deal with it, but if we have to wait on Psoft, we'll get out of here.

The way you treat your customers is totally inadequate. Yes, you were hacked, but servers are never safe. With or without ssh. You already have some form of protection because you choose who you give ssh access to. You could stop giving ssh access to any new customers, but to remove it for all your customers is just way out of line.

Winboxes are so easy to hack and you still add them monthly because people without any hosting knowledge feel safe to use them.

[jmbeach]

Originally posted by somereseller
Winboxes are so easy to hack and you still add them monthly because people without any hosting knowledge feel safe to use them.

Hey, watch it! I'm a Windows host, who isn't just some schmo - I also use another Window's Host who I've never had a single issue with in 3 years. Not a moment of downtime that I ever saw, no hacking issues, nothing, granted the single account there costs the same as a reseller account here.

Let's not start a Windows/Unix war here. Any machine can be made [relatively] safe with the right administration.

I'm with you, though, on the across the board removal of features users have come to rely on like ssh - let's just hope that they can come up with a compromise that protects everyone while still giving access.

[Vixen]

Well, maybe we need to do what we used to and make everyone that wants SSH access fax us a copy of their driver's license and sign a form making them responsible for the access they are given.

Because it is obviously becoming an issue with allowing anyone and everyone to have it.

[jmbeach]

FYI, you guys should definitely remove SSH from your matrixreseller website, so that new signups aren't misled.

Just a heads-up if you do plan to disallow this.

[Vixen]

I think they are more worried about getting everything fixed right now. However, I will point that out to Admin later.

[somereseller]

jmbeach, my intentions were not to say that every windows' user was a schmo without any knowledge, but it's easier for somebody who can use a winbox at home to get started with windows hosting. Some people get started without any knowledge just because they think that they can make some money without knowing too much. GUI interfaces help them a lot, but in the end misconfigured scripts on a full of buffer overflows system can be very dangerous.

[admin]

somereseller, it may only be fore a few days we are not sure yet. I can tell you this much if its turned back on it will be jailed SSH only.

90% of the stuff you can do from SSH you should be able to do from cron. We are wroking very hard to get the mess stright and if you have to go I am sorry to see you go.. But hey we don't want hacked systems on our network and will do any thing we have to to stop it.. If you don't like that then hit the door as that will never change. We want our systems as secure as they can be for us and you..

Whats easier to tell your customer:
1> SSH may be disabled for a few days
2> Your site has been hacked and all of your credit card numbers stolen.

I think every one here would pick #1 any day is why we are doing what we have to now to make sure it CAN never happen after what we saw at 3am today.

[mresell]

I also depend on ssh. Realizing you are trying to keep servers secure is it worth it to eliminate a hugely important feature because of a hack. There really is no 100% security. Offering something then just changing it with no notice is not good customer svc IMO. I do think that since it was a new customer disabling ssh for everyone is not logical. Just don't approve new ssh until you can make ssh more secure.

[somereseller]

Vixen, we already have to fax some form just to get a plan, I see no problem with signing another one to get SSH access.