7/19/2005 NT12.hsphere.cc

[Vantage]

NT12 has become unstable. It will not stay up for more than a few moments.

It is being brought back to the office. I will post more details after we take a look at it.

[Vantage]

At the moment it looks as if the Raid card in this server is not being accepted byt eh OS anylonger. We are working on updating drivers..

[Daniel]

NT12 is almost backup and running. No data was lost. We will keep it in office once we have the machine online for monitoring. Then after a period of time we will move it to the datacenter, mostly likely in the evening times.

[dvanburen]

The old hardware has proven to be unstable. Unfortunately this has delayed the re-deployment of the server. Rather than try and troubleshoot the old server, we decided it would be faster to move NT12 to new hardware. Rest assured we are working as fast as we can to get this issue resolved.

[dvanburen]

Despite using Raid 1, we are going to have to restore all the data from our backups. The latest backup was last night, so there should be very little difference between the data we are restoring, and what was on the server when it failed.

NT12 is on it's wasy to the datacenter now. As soon as we have it installed, we will begin to restore the data. Once that is completed, NT12 will be back up.

[nlane01s]

Any estimates on time? I have a very important client who's site is down and is basically all over us for this issue at the moment. I need to tell them something on the status and when it will be back up.

Thanks!

[Phydostyle]

We have been getting hammered from our customers as well. Since we will all be, most likely, offering our customers discounts on next month's hosting charge, where does that leave us?

It' been over 8 hours so our SLA does apply for a free month credit on your account. You will just need to fill it out after the downtime is done.

[scarlettsweb]

Hi! I know that i've called a few times, and pestering constantly on this one, but i'm trying to be patient I have a client more impatient than I am, so I have to keep trying.

I know that it's up- however, the HSphere CP WebShell, and FP extensions aren't working, the site is not on the server- but login.html is.

Is there a new ETA on completion for the NT12 Server, and if so, will we need to republish sites? This client has a secure server, dedicated IP, external DNS, etc., and I have to be sure they are going to be 100%-- or i'm seriously in the doghouse.

THANK YOU for any response, and for all of your diligence with this one!

CP is done it's part, files are restored setting permissions now. Sites should be coming up now.

We did our best to fix the raid 1 ( YES THIS SYSTEM HAD AND IS RAID ONE AGAIN FOR THOSE THAT WILL ASK ) but that did not turn out to well.

This is the 2nd oldest system in the cluster NT11 being the oldest NT system. NT12 is now a new dual xeon system with 2GB of ram scsi system. So the downtime sucked but the system got a BIG upgrade from it's old self.


Once permissions are done we will close this post so and use tickets for this system if you have issues. Even at this point if you have an issue you should open a ticket because we maybe able to fix it since all IIS, User and site info is now on the box.

If you had custom DLL's or SSL installed please let us know. The old raid 1 box kind of boots, so we might be able to get that SSL cert off of there to reinstall. I did not have time to check for those when fixing the box. Sorry just wanted to get it back up.

[scarlettsweb]

Great job

[Brangwyn]

We did our best to fix the raid 1 ( YES THIS SYSTEM HAD AND IS RAID ONE AGAIN FOR THOSE THAT WILL ASK ) but that did not turn out to well.

Raid 1 Doesn't offer really offer the greatest of redundancy it really only protects against hard drive failure, and even then getting windows to boot of the secondary complex drive isn't allways straight forward .. nor if it's an actual raid card failure I guess

Good work getting things running again anyway

Yea I tried. At first it was just a drive, but after getting it to boot it went down hill from there. All coming back together now though.

[Phydostyle]

I did a lot of work on one of my client's sites Monday during the day, and now everything I did is GONE. All of the changes I made. I spent about 6 hours making updates and now I have to redo them all. There is no way that a complete backup was done last night. If that were the case, my changes would have been restored. These were changes that were promised to the client by Wednesday. It is now 2:45 in the morning on Wednesday and I have to stay up the rest of the night redoing what I did on Monday.

[Brangwyn]

You may want to consider changing how you develop or maintain your customers sites. Developing directly on the live server is bad for several reasons, not the least that a) you end up not having a local working copy, and b) your effectively "testing" changes on a live server which may impact everyone else using that server.

You may want to ask techs via a trouble ticket to double check your site was restored correctly, also bare in mind the time of day when backups are made will effect just how complete the backup is.