01/16/2004 Mssql2

[Brangwyn]

The drive was my first guess A low level sector check and then any one of the tools around for stress testing the drive should have pinpointed it farily quickly.

IBM Drives are regularly not IBM Drives, they (now at least) buy the physical drive from several of the top 3 makers (Seagate, Maxtor, etc) ... yours probably had a crappy Maxtor in it

Second guess would have been swapping the memory.

We're not sure that it was the drive yet. We're just operating on the principle that any IBM drive found in production gets replaced at first opportunity, period.

[Brangwyn]

So what are the actual symptoms of the problem then ? anything other than just SQL Server marking a few DB's suspect ?

[Maayan]

So, what you are saying is that you don't have a replication server that would have solved this miserable incident long ago, and you have lost 12 hours of our valuable data?

[Vixen]

I think what they are saying is they are installing a new hard drive, reinstalling all the software and then restoring from backups. But I could be wrong, I guess.

I understand everyone is frustrated but yelling and complaining is not going to get it done any quicker. Trust me, I am not happy that my husband is spending Friday night at the data center working on this server but things happen that can not be predicted.

[Brangwyn]

Originally Posted by Maayan

So, what you are saying is that you don't have a replication server that would have solved this miserable incident long ago, and you have lost 12 hours of our valuable data?

What makes you think they would have replication ? Just by looking at the SQL Server via EM you'd have seen it didn't have any replication partners, and they've never advertised that they do. If you want that degree of redundancy you'll need to find specialist hosts who do provide it (who no doubt will charge you a bucket load more per month than your probably paying here).

Vortech have always been pretty consistent on backup polcies as well, its up to the individual account holder to ensure they have satisfactory backups to recover their data if required.

All the non suspect databases are in place and the suspect databases are being restored from backups...

[Brangwyn]

Any chance of a list of the suspects ? I know thats a big ask of ya'll, but it would help tremendously if we were able to determine which specific customers of ours have been impacted (though I'll probably be crediting all customers using this server anyway). I'd just like to be able to advise those of mine who have had data loss due to the restore rather than them advise me of it

MSSQL2 will be going down for a short time while we put the case back together and throw it back in the racks. Restoration of files is complete.

[Brasil]

PLease, keep us posted.

Much of our business comes from a client that is down.

So..., boring as it might sound..., I have already given up my friday night to keep up with what´s going and its resolution.

Thanks very much

[Brasil]

Any news guys?

When it´s gonna be up?




Thanks very much

It's up. If you couldn't connect to it, it may have something to do with your configuration. Please note that the server's old IP address will be (was) dropped. That is, 216.157.131.30 will very soon be no more. We're bringing it back up for the stragglers but it won't last. Check your global.asa files and anywhere else that it might be, and do the same for other SQL connections while you're at it. Mssql2 is at 216.157.145.62.

Before someone goes ballistic over this, we announced the move of all H-Sphere servers' primary IP address to the 216.157.145/24 netblock some weeks ago. People should not have been using the IP address anyway--that's why we have a DNS and give everyone server aliases.

[jjay]

Just want to say that my databases that were on MSSQL2 are working now. They hadn't been since it originally went down but now is. (I'm pretty sure I was part of that 30% or so few who were part of the "suspected."

*Also you guys do and continue to do excellent work! I'm very satisfied even if my customers aren't that a database server was down for a while. Fact remains that for what we pay versus other hosting companys Brad & the rest of the gang still goes above the call of duty and usually in the end gets things back the way they were. Let's face it nothing in hosting or the internet world has 100% up time and probably never will as long as people want more\better\upgrades and impovements.

The very fact that you were able to restore instead of having us restore makes me happy.

Good luck to whoever is still having trouble! *I honestly know how you feel but keep the faith in the Vortech Team.

[Brangwyn]

Thanks for your work guys

So it did turn out to be the drive then ? or was something else required to kick it into life again ?

The filesystem was corrupted. We're not sure whether it was software or hardware failure since we haven't run diagnostics on the pulled HD yet. We'll do the post mortem tomorrow or Monday.