nt38 11/28/05

[Lesc]

Hi Guys

Is there any further updates with regard to getting all the permissions, etc restored?

Thank you

[jjay]

Originally Posted by byron

The initial file and directory restoration from backup process has completed. We are now running scripts to restore proper ownership, NTFS permissions and ASPNET settings to all directories. This could take some additional time.

what is the status of this operation?

[mresell]

I don't mean to jump on the band wagon, so to speak, but this does keep reoccurring. The backups being corrupt has happened in almost every case. Drives failing is the pits, but there has to be plans that work to limit this. When it does happen deal with it effectively and to work around it. What ever drives you have used there have been these kind of issues so I don't think you can readily blame it on the drives per se, especially since you seem unclear why they failed. Seems like diagnostics would give some indication. Not sure what you are using for hd monitoring(only nagios?). I understand cost is an issue to a point, but you have to admit the enormous time it takes you and the loss of customers IS money that you are spending.

[badchauncey]

I've lived through 2 "failures" where data had to be brought back. 1 was "sorry, all backups were corrupt" and other was this one, saying 4 days old. So it's safe to say that backups are very unreliable. But if you read over the terms of what we are "promised" as resellers, backups are not one of them

[supportsolution]

Originally Posted by Brangwyn

With the revenue you loose each time this happens, hiring a certified disaster recovery consultant to come in and take a look at things would probably pay for itself pretty quickly. Someone who knows all the products out there, their capabilities and shortfalls and who can suggest and implement a failsafe BRP strategy for you.

Strange, I do that. Been doing that for the last 4 years. But then again I have not posted to this site often enough to warrent a fair comment lol

Not to worry though.

Dave.

[pentom]

Is there any update on the permissions being restored ???????

A couple of our sites are still down since they rely on updatable db access.

An update would be appreciated.

[admin]

Well the good old psoft permission script died at I something. It has a -s option to start at the last user but of course that does not work. Grrr... So we had to start it over and opened a ticket with psoft so ether the script will pass where it was or psoft will answer and we will get going from I again.

Not much we can do to make this any fast and really wish I could..

[Brangwyn]

Originally Posted by supportsolution

Strange, I do that. Been doing that for the last 4 years. But then again I have not posted to this site often enough to warrent a fair comment lol

Heh don't worry .. I've got certifications coming out my ears and I'm not always agreed with here or in my day job either .. you can't be right all the time I guess (though I suspect there's a certain type of personality that enjoys learning things the hard way too, I know I do myself sometimes).

[admin]

We found the issue with the psoft permissions script and have it restarted from where it left off so it's still going along. But all sites should be working, I heard a customer having an ODBC issue on the phone today, a simple way to fix that is to remove it from the CP and readd it. We found it skipped a few system ODBC's..

[admin]

Also ifyou guys have a better idea for moving data bring it on, show me something that works. I am all ears but no one has ever been able to show me anything that can moved millions of files over a netwrok very fast.

FYI I have tried most of the windows backup software, ARC, Windows Backup, anything you name it I am sure I eather looked at it and it just would not work in our case, was slower then we used now or was just junk.

Our system is rather simple and works right now, data is backed up using robocopy over a secure connection to backup systems that can't be accessed from the out side. To move the files back we use robocopy. Here is the down fall, that means we have millions and millions of little files and what takes SO SO long to move back over. But it takes just as long to zip it up on the back up servers. We thought about 1 day backup 1 day zip so they would be pre ziped, but I just don't like that idea much and a backup server can zip one of the web servers in under 10 hours and each backup server holds 10 to 15 servers day backups and weekley backups. We have the space but no time or way to do it.

Of couse moving one huge file would take 1/10 of the time but then you also have the time to extract that file once over there and well you are reading and writing to the same drive again it's going to take for every again.

Any ideas?? Like I said I am all ears and I do listen to you guys. Also supportsolution I was not picking on you, I just did not know who you were.

I don't ever clam to know everything, but I have tried a LOT of crap in over 5 years on backups and nothing ever seems to be 100% or fast in that matter.

[craigrwarner]

Do we have an update on perms.. this downtime is getting extremely lengthy... customers are not happy. Thanks.

[badchauncey]

If you don't know a faster way to restore from failures then I guess it will be up to us to decide if 24 hours of site downtime followed by another 24 hours of not able to update the site or use Access databases is acceptable. I got my credit today, and the problem isn't even over

[Brangwyn]

Brad, the issue isn't just the speed of the recovery, the validity of the backups seems to be questionable pretty often as well, No-one has confirmed in this thread that I can see, but earlier posts seem to indicate this servers been regressed about 3 days becuase recent backups weren't available?

The method I suggested earlier where you backup initially to a drive on the same machine works quite effectively, then you can use Retrospect or any other product of your choosing to ship another backup off to your main backup storage JBOD, leaving you normally with 2 good copies of your data at all times. You can spin down the second drive in the machine to preserve it's life span as well, further mitigating the risks of having two drives having near simultaneous physical failures.

One reason I like Retrospect is the compression you get both backing up and restoring, this does cut significant time from your restores becuase instead of transferring 100GB you end up transferring about a 5th of that on average.

Rather than using RAID1 which has it's flaws (though it does have some benefits as well like improved read I/O .. hard to measure though becuase webservers do tend to have higher write I/O's than you'd expect and Raid 1 write throughput is generally accepted to be slower than that of a single drive configuration) Using 2 drives per machine not in a mirrored config (for me personally) has served me pretty well.

The biggest problem with most forms of Raid as I mentioned earlier are that if you loose the controller, have software corruption (virus etc) then you're still left in a total recovery situation, you can't rebuild from the array becuase both copies of your data are corrupt.

[kapuwa]

We are with Vortech for the past two years and with all these problems we will still be with you. I tried three hosting companies before coming here and found that this is the place to rest.

But, the big BUT

We are very disappointed to see that the same problems are coming up again and again (although promises from staff at end of the each disaster that you have taken precautions). I think month/two ago, we had a much similar problem with a UNIX server and we went trough the exact same thing. You even went back to the old backup system saying that it is much more proven, but, looks like that also failed here yet again. I strongly believe that backup you used to restore here is not the latest because, I got customers reporting loss of logs + recent changes happened over the last 24 hours prior to the crash. Most of the people here will agree that at Vortech, everything is EXCELLENT except disaster recovery. Many will agree with me that 2 days to restore a server is totally unacceptable. Anyway, what ever happened is over

NOW, my two cents goes

(1) I understand what Brad says – we can get the moon and the sun for $35/month. At the same time if we can not get a site back to live after a disaster in at least one day, whole thing does not make any sense. Now, I am sure, there are people here will pay more for bit more reliability than this. Can you guys come up with more reliable servers, increase the price or make a new plan. So, we can put the most impotent customers on that server and pay you more. So, it’s a win-win situation.

(2) Customer base of Vortech is growing pretty fast and we see CP error more often now than the past two years. Is it the good time that you start a new cluster?

(3) I strongly believe that backup/restore system at Vortech needs the TOP attention. Over the last 2+ years, I have seen at least 3 smaller disasters and at each case – your backups didn’t help you at all. I am sure; there are lots of people with experience regarding (excluding me) these things. I suggest we open a new thread and do the “TECH TALK” find the best possible way sort-out this issue.

I understand that no hosting company can be PERFECT. Tech support team at Vortech is excellent EXCEPET when it comes to disaster recovery. I believe this is the worst thing happens for us as customers and to the Vortech.

My only hope is that Brad and the geeks will go to the board room and come up with a nice plan once this episode is over to prevent anything in the feature.

I wish you guys’ good luck with your efforts.

[admin]

The C:\hshome is setup so that all sites should be working, we have a temp fix with adding everyone to the hshome dir until psofts script is not working along with asp.net if your access DB is not working please open a ticket or if you opened a ticket but did not get a reply that you think is right or it is not working still please PM me that ticket number. I will answer you my self and tell you if it's server or something else.

We have a meeting tomorrow and I am going to look in to putting 2nd drives in every system we can right now or system with 2 drives in a mirror breaking that mirror and using that 2nd drive as a lock first content back up, then we would pull that backup off to the backup systems. This would reduce our content transfer time some but to be honest not much. When we replaced the drives in NT1 NT2 and NT3 the other night I think it was we copied from old drive to new drive this still took 8 to 10 hours.