Nt20 8/28/2005

[kapuwa]

we are on which letter(s) now?

[admin]

Mailing I just sent out:

Hello NT20 Resellers,

Due to some problems we have had during the restore process of NT20 there are a few things you need to be aware of.

1) We were forced to do the restore from our weekly backup from 8/24. If your site was updated after 8/24 you can upload it to the latest version of your site once the restore is done.

2) Some sites were not restored due to the difference in timestamps on files and what we had from our backup. If you find your site is missing, please submit a ticket and we will try and manually pull it from the backup system. Include your username and the missing sites.

3) We put a default index.html page in everyone's home directory. If the restore did not overwrite this file, or you use a different directory index, you will need to set the directory index manually in the user panel.

We apologize for all the inconveniences this has caused. We are working very diligently to improve the accuracy and efficiency of our backup system. Due to the recent events we have decided to double the retention time of backup volumes to ensure the most complete restores possible and to avoid trouble in the future.

Best Regards,

Vortech Staff

[admin]

We are on T now. You may upload your site if you wish, if it's not there then we can try it by hand of coruse free of charge.

If we do not have it you will neeed to reupload the site or tell your customers they will simply need to reupload or restore from back up. I hope some users made use of the new backup option in the CP.

It seems everytime a restore comes up something just has to go wrong for us. It's getting a little old for me and I know it has for a few of you guys. We have over 5TB of back up now all on raid5 so to solve the problem we will be taking that 4TB to over 8TB here soon and double the backups we keep on hand for restores.

[kapuwa]

I belive most of the things at vortec works great except the backup and restore. Everytime time when there is a disk faliure we restore from tape backups, does that means no RAID on most of the servers?

We all assume that you do backups every 24 hours but in this case you went back so long and used a 3-4 days old backup to restore. does that means no daily backups? We experinced the same when the mySQL box failed few weeks ago, we had to go back about 4-5 days back.

Althought i am happy customer of vortec, this is something Brad and the administation staff should look at ASAP. If you can get the backup and restore sorted out am sure Vortec Rulz!

[Brangwyn]

Brad I don't really think the issue is how many TB of backup storage you have, the issue as I see is it that under the current backup/restore process it just quite simply takes too long to recover (for whatever reason).

I personally think your strategy needs to be reconsidered. Instead of relying on a massive backup array you should be spending that money on redundant drives within each server. As I understand it you have most servers running mirrored system (OS) drives or something to that effect, why not just move the entire server content OS + User data to Raid5 (3 Drives + hot spare) or similar instead (even software based raid 5 parity would be better wouldn't it?) if the entire system was Raid5 and a drive failed the server won't even skip a beat, you can then replace the faulty drive at your leisure.

You may need to move website logfiles to a non Raid5 drive for performance or just use 1+0/0+1 instead of raid5 which I guess is really only a good solution if 90%+ of the disk access is read only.

[mresell]

This is a real problem. The performance should be negligible for raid5. Time you and customers are spending is costly, not to mention lost data. The fact is HD's fail all the time so why go thru all the headache and risk.

[Brangwyn]

Raid5 isn't a perfect solution for a webserver mresell, think of all that writing it does to web logs, thats bound to cause a write bottleneck. If you have the logs outside the array though, it can be a good compromise imho.

[admin]

Well each system goes out Raid1 in most cases now. NT20 is now Raid1 73GB 320 15k drives. It's still costly for raid5 in the web servers the way the get eaten up. Even though it nice we are adding boxes, just because we are adding boxes does not mean we are making all that much more most of the time. Adding 2 more drives plus a good raid card is driving that server up about $1000 even a our cost. Most of the time we do 3 to 5 systems a month thats $3000 to $5000 per month.

Something we are looking in to at least for "OS" drives is maybe a SAN setup running all the OS's and running on Raid5 with a mirror SAN of that one. This would give the OS drive a much high up time. Then we can do the home drives in raid1. This may be the cheapest and best way to go. It's also cheaper in the long run after the first 10OS drives are on there. Plus we may be able to move current systems right in to a SAN.

But still going to get the back up system fixed fully, we just switched to the new system bacula, its better in some areas then our old system, but has let me down 2 times already since we started using it. But I think the issue from that is now fixed.

[Brangwyn]

Whats the status of the restore currently? I've still got a "K" customer with nothing seeming to work, I can't even login via File Manager to check if the content is restored.

P Q and R are restored.

[Vantage]

S is done

[levseltzer]

Is there some way for resellers to quickly see a list of which clients are on NT20? I see I can list each client via the Search function, and then click on the account ID. But I've got dozens of accounts, and it takes quite some time to do this. I don't want to send a message to ALL my clients again (like I did regarding last week's problem), as it causes unnecessary concern and lots of support time explaining to most of the clients that they do not have a problem.

Thank you.

[powvex]

All my sites on NT20 have problem. Over 15 hosts, and all of them have different problems:
ACL error 403, NT20 page showing, or not restored.

for god sake not even one of my sites on NT20 is working right.

What are you doing at vortec?? what is the meaning of RAID backup? what is the backup option on all servers being done each day??

We are now being abuesed by Tel/Emails from our customers, and I know they are right about this problem. Many of them dont have any backup on their sites as they are dynamic.

[Vantage]

All users are done.

Some users will not have all of there information restored due to the issue with the backup client. We are working on this but in the mean time please submit a ticket if your data is not complete.

I am resetting permissions now. SOme people will see all there data some will see nothing. Please do NOT submit a ticket yet if you see nothing. It means that your permissions have not yet been reset.

[kattouf]

What about user who signed up after 8/24, would their accounts/folders be there? i suppose not...

All in all i have to express my disatisfaction with the restore procedure in place, it takes far too long. I now have signups everyday, and lost revenue because my service domain was on nt20 :-( .

But i am happy here nonetheless.