Nt49 08.08.2007

[dvanburen]

This should have been started hours ago...I apologize.

NT49 is down and being worked on. We hope to have it up soon. Please check for updates here rather than submitting a ticket. Speaking of tickets, there will be some long delays in responses to tickets and phone calls will go mostly unanswered until this problem is resolved.

[murdok]

What is the nature of the problem being resolved? - server down nearly 7 hours now according to Nagios monitor.

[murdok]

Any update you can give? Nagios is not showing NT49 problem anymore, but still cannot access my domain "jht.com" or FTP to it.

[dvanburen]

NT49 is back up on new hardware. I had to go with the file based backups. They are being restored to the new server at the same time the domains/users are being recreated. Some domains already work, many do not. It is still going to be awhile before it is 100% done.

[illustratedlife]

Does this mean that everything since July 29th has been permanently lost? What about nightly backups?

[jmbeach]

yes, it would be good to know if backups are now only done weekly. I just had a client on NT49 ask why his changes from a week ago reverted to the previous version... I guess I assumed that backups are done daily.

any way to get notified as a reseller when any clients with data on a particular server are affected like this?

[illustratedlife]

At least FTP is finally working again. . .

[dvanburen]

The recreation script has finished. The content transfer is about done. Once it has finished I need to tighten down the permissions and we will be about done. It looks like SSL certificates will need to be reinstalled unless the old drive decides to cooperate (hasn't yet). Application roots also must be recreated, email support for this.

[dvanburen]

The controller died in the old server and took both the OS and the image drive with it. We used the file based backup that was just over a week old.

[illustratedlife]

Please keep trying to recover that data. With a web-based business, it's incredibly costly to lose 10 days worth of customer data (and orders)!

For future, wouldn't it make more sense to do your backups on not-the computer you are trying to protect? Seems to me the point of backing up is to have another copy in case of catastrophic failure, which dictates that the data should be stored somewhere where it would not be affected by that failure.

[dvanburen]

That's what the file based backups are for. If you lost 10 days worth of customer data I am guessing you are using access. Access is ill suited to a web environment, you should consider migrating to a more reliable platform such as MSSQL or MySQL.

Also keep in mind that it may seem obvious to you to keep them on another server, but the likelyhood of actually losing both drives is quite low. Unfortunately as we have just witnessed it does happen. We have recovered servers many times, often in less than an hour from the time we are aware there is a serious problem, specifically because the image is local and does not have to be streamed over the network.

[dvanburen]

NT49 is done unless I happen to be able to pull any data off of the old drive. Even then I do not believe that I will be able to get it to boot, so no SSL certs.

Here is a list of what was lost:

1. SSL Certificates
2. Application roots
3. Custom Permissions

[illustratedlife]

Personally, I also prefer mySQL. The decision to use Access was not mine, and we have not had the opportunity to rewrite the site to incorporate mySQL.

The Access databases are not the only customer data we had stored on there. It was also our central repository for client proofs. Our artist gets to spend her time finding 10 days worth of artwork and reincorporating them to the site.

We understand that you have no control over hardware failures. Please understand our frustration that we lost 1/3 of this month's data when we thought our host had robust nightly backups. If you can get any data back, it will be very appreciated.

[Brangwyn]

any known issues with FPE still on this server? one of my customers just had the Frontpage icon disappear from her control panel.

[jmbeach]

dvanburen:

so can we have clarification how often in-house backups are done on the webservers? Are database servers more often backed up, as you are implying?