nt38 11/28/05

[CreativeNRG]

I hold my hands up and admit I know relatively little about hosting but I am not a technical novice, I therefore require an explanation that I can go back to my client with.

I think you refer to it as redundancy, but why is there no system in place that allows websites to quickly be switched over to another server should one of them go down(e.g. a mirror), therefore allowing minimal downtime for the sites that were hosted on that server.

What am I supposed to tell my clients who are more than likely going to bring up the point about the potential money or loss of business that is the result of this extended downtime which is a number of hours now ?

I apologise for hassling you but if I get hassled by my clients I will in turn need to keep asking you information to go back to them with as you are the end service provider. I have been very happy with my hosting service up until today which has caused me to lose confidence, not because of the problem but because of the length of this downtime which as advised is still going to be a whole lot longer.

Thank You

P.S. BTW Vantage, your email signature is very appropriate

[tkraffty]

ouch - and I was cooincidently planning to add monitors for all the sites we have on this machine (about 40) to our internal server monitoring system (ServersCheck) today... damn gremlins...

As soon as you can post an ETA on recovery - and let us know if data from any time-period was lost in the process - it'd be appreciated. I know it must be crazy there right now...

TK

[Craig_Smith]

CreativeNRG, Ironically this machine was setup with a 'redundant' RAID 1 configuration and 2 drives. Imagine our surprise and frustration when both drives died. This type of stuff is really inexplicable and when it happens which as you all know is not very often, it just sucks.

We've got everyone working on it, and called Dean in to oversee the restore process. We estimate an approximate 8-10 hour time frame to finish this. We'll post here as we know more.

Alright I just received new information from the Tech rebuilding the server. The problem was that both drive failure and were unreadable. So the redundacy we have was negated due to both drives going south. We have rebuilt the array and now are transferring data to the new drives. The amount of data to transfer on the drives will take a significant amount of time yet. We will be posting to keep you all in the loop.

We do not have an ETA or further information. When we do we will post as we understand you would like this information for your clients.

[badchauncey]

The devil might be in the details. I am seeing reports of "mysterious" disk problems on NT 38 on 11/24, 11/26 and even earlier. wonder if that was a sign of things to come (that came today

[Silverbug]

Thanks guys, keep up the good work, and let us know when you know more.

[Craig_Smith]

thanks for your patience guys, we will update again ASAP.

[jjay]

what is the latest status on NT38? and expected uptime if one?

Rick

[techaspects]

I am just curious on the update status. Please keep us in the loop so we have something to tell the customers when they call us complaining. Thank you for working on this issue so hard.

[dvanburen]

The data restore is up to the letter 'D'. There are some large accounts, so it is difficult to say what percentage this actually represents. The account recreation is about 40% done, and should finish well ahead of the data restore.

[CreativeNRG]

So basically we are looking at a good few hours yet, this is going to be a 12 hour downtime!!

[badchauncey]

When you guys say "D" for restore, do you mean the reseller name?

just curious, I am an "M"

[Brangwyn]

The chance of two drives failing at a physical level at the same time is probably on par with your chances of winning the state lottery. It would be more likely that there was corruption caused either by software, or at the controller level, software corruption being more probable of the two especially if you've managed to start rebuilding new drives without changing any additional hardware.

And there lies the weakness of virtually all RAID Level redundancy, if software or a virus decides to cause corruption, the corruption gets mirrored as well taking out your entire disk subsystem with it.

After the problems dealt with, It would be interesting to try and mount one of the drives in a *NIX machine, you may find the data's recoverable that way.

When you guys say "D" for restore, do you mean the reseller name?

User Account Name not reseller I believe.

[techaspects]

Is the Letter you are restoring based on the account holder name or the domain name? One of my domains in question start with "C". Just curious.

[admin]

It does take a long time to do a restore, we do the restore of content and IIS/Users at the same time.

Also when we say "D" that means the the end user name not the reseller name, that has nothing to do with the restore really.

From the looks of it the restore of content might be done before H-Sphere is done with it's part. But it really does not matter since we have to have both to be any good, IIS with no content or content and IIS not done yet is not much help.

This was a raid1 system and the raid funny as it is was checked on 11/26/05 the raid showed good and both drives showed good. This AM both drives came up bad 0kb and should have been 250GB.

This is the 3rd time now raid has failed us in such away. Ether both drives fail, one drive fails but then it can't rebuild it. Grrr. BTW this was hardware raid not software raid incase any one wants to know.