|
MOST sites should be up at this time. I did say most, it looks like about 90% are successfully re-created. Some sites are causing problems, and as I stated before, SSL's will have to be manually re-installed. If you have a site that is still down, Please submit a ticket with that site's information (domain name, account user name) and we will try to get it up for you. In some cases this may require a head administrator to see to it in the morning.
To answer the question of what we're doing to prevent this in the future:
We are Installing and rolling in new backup software. We're changing our backup strategy. Logistically it's hard to balance speed and server load with space and ease of restore. Anyone who used to complain of slowdowns at the same time every day but does not see them anymore has already seen the benefit of new software. We have to be careful when we make changes, not all servers have been updated (this unfortunately was one that had not) We've tried some backup solutions that bring a server to it's knees as soon as volume gets high while backups run. Backing up files in the background to another server is the least intrusive, but offers the slowest restore (it's what we're going through here) Imaging a drive offers the fastest restore speed, but few software solutions throttle well, overwrite older backups automatically, can be deployed and managed centrally over multiple subnets, compress efficiently and use little cpu resources. We're searching now for that Ideal mix where we would have an image file (fast to restore) which doesn't affect performance during it's generation process, overwrites old images, while keeping our old emergency old standby of copying files off weekly also. It's a lot to plan out and make work with the best use of all resources. Especially considering there is no room for error. Down time is bad for business, but a backup solution that fails and when needed is not there, would cause content losses that would be even worse!
We are open to suggestions. Some of you that have been here a long time have seen us implement strategy and scripts based on suggestions from customers. I personally like hearing feedback on what all of you are doing. I've seen some really entertaining and well made websites from our customers. I like to hear what works and what doesn't work from you all. Please don't think we're ignoring your plea to make sure this doesn't happen again. I REALLY do not want it to happen again! We've only had a few serious server crashes, but every one of them means long hours, no sleep, and lots of phone calls being yelled at with expletives in 15 different languages. Believe me, I want things to stay up, I want machines to be reliable. My goal is to get more new hardware in place, cycle out some older machines, and try to get far enough ahead of the curve things like this never happen in the first place (if all went well backups would be a never used safety net). In short we're doing what we can to keep you guys happy. Please understand when there are limitations to what we can and cannot do!
|
06-25-2008, 03:49 PM
Linear Mode
