MAIL9 Disk Maintenance

[Andy]

We are checking the disks for errors and checking the disks for performance on the MAIL9 server.

[Andy]

We are not satisfied with the results of our testing. We are replacing hard disks on MAIL9 to avoid future delays of mail service. We will update again when we have an exact ETA.

[3WHosts]

Hi Guys
Any idea of an ETA please? My clients have been without email all morning. I need to give them some info please!
Thank You

[stanj]

Thank you Andy, I've got the office placated by using an alternate G-mail account for urgent messages. We are in the middle of a a work day here, but you are working in the middle of the night.

[server-68]

My european customaer are without email for this issue.
Why is not possible to have a disk backup to use in this cases?
I mind, are almost 4 hours without service...and if this is not a problem in USA, because happened at your 04:00 AM, is a problem here, when the companies are open!!!

[3WHosts]

Hi Server-68
That is exactly my problem - My clients are in Ireland and have been without mail all morning. It is causing serious problems. The only update that I have had from support is that it will be sorted as soon as possible. Not something the client is happy to accept!

[Edwin]

Hey guys,

We put new hardware in the rack and are now working on copying everything we can back over. I'm not sure of what the exact ETA is but we will let you know when everything is finished.

[mbz-bugsy]

Hi,
Well the reason why they do this at this hours is because they prioritize the US customers more then Europeans. I’m in the same situation as you. All we can do is to pray the American god they do this quickly as possible. :P.
Now who said that life was easy?

[server-68]

Why not have a backup disk? Why noy? Too expensive?

[jtaylor]

Why was this done so close to the start of business, and obviously already into the start of business in Europe????
Clients are screaming, we should have been forewarned about this BEFORE it was started at 4 am. Normally, maintenance and checks are done in the middle of the night or over the weekend to avoid this situation - so why was this not handled the same way?

[desertdirk]

still cant log in
DD

[JoshK]

I'll attempt to answer all the questions in one shot.

You cannot run a backup copy of mail. It is volatile storage. Backing it up would be futile as data on the hard drives literally flows through storage it is not kept for long periods in most cases. The speed loss of writing backup copies would degrade mail service significantly. As sits we put the highest RPM drives readily available in these machines just to keep access times down. We do have a backup of the OS drive, but that drive sees little wear and hence rarely fails.

We assess hardware when we notice there is a problem. Would you rather I wait till the start of your business day, but then have the server down for two or three days while we configure a new one? This was not scheduled, I saw errors, took it off line in hopes of a quick fix and instead saw hardware issues. While the machine was down, we slid a new machine complete in it's place. This was to prevent future problems.

As to location, we do schedule most things in the middle of the night in our time zone. That is because statistically there is less traffic. We watch the total numbers and attempt to schedule all tasks that might incur machine downtime in the times of least traffic. When possible we schedule planned take downs for the middle of the night on Saturday here, because the total worldwide traffic is the lowest then and it is not buisness hours anywhere.

To repeat though, this was not a planned take down. The machine was displaying errors, and when we checked the disks the error quantity was too high. When I ran performance tests the disk was failing. At that point it was replace it now and have limited downtime, or risk loosing all mail and possibly days of downtime. I must do what is best for all customers involved. A small delay now is better than days of downtime and lost data later.

Currently new hardware is in place, we're copying all the data off the old drive to the new drives. We expect it to be fully functional soon. It may be functional for some customers now as the directories are restored.

[3WHosts]

Thanks for the update and clarification Josh

[jtaylor]

If it was an unexpected occurance, then I can understand. However, you guys might want to put something to that effect in the first thread posted about a "maintenance" so that we know this and it doesn't appear that it's normal checks that could have waited or been done earlier (in the middle of the night). Some of us are tech people and can understand when there's a true issue - give us a little credit. Otherwise, we will be assuming that we are just getting yelled at by our clients because you guys didn't warn us about a maintenance time.

[rheide]

Josh,
Thanks for the update. Can you estimate the definition of "soon"?

Does it mean 30 minutes or 5 hours? Just a ballpark would be nice - I won't hold you to it, but at least we would have a rough idea.

Compared to next week, 5 hours would be soon. Compared to noon, 30 minutes would be soon. :-)

Thanks!