2008-04-23 - Mail 6 Outage

[Edwin]

Hello all,

I apologize for not putting a notice up sooner, but as many of you have noticed, Mail 6 is down and we are currently working on it. We will be posting updates here as we get them. So far I believe one of the drives decided to fail, but we're checking it out right now.

[dickkyboy]

Hi There,

Is there an update on this?

Has been down for some time now.


Thanks.

[mdallarosa]

It seems to be backup now with no data loss.

[JoshK]

Mail6 was brought back online around midnight EST, has shown no problems since, and it appears to have caught back up with little or no lost data. There may be a few messages bounced or lost due to the time the machine was physically disconnected. We are sorry for any inconvenience that this may have caused.

[zubin]

this server seems to be down again. no ping.

[chungmike]

Yes, it's down again. What's up with mail6?
I was hoping reliable mail service would be one of the improvements implemented after the new reseller pricing deal.

[Brangwyn]

Wasn't it suspected to be a drive failure 2 weeks ago when this last happened too ?

[JoshK]

We hope to replace the drives in Mail6, as we occasionally see errors in their file systems. We just have to work out the logistics so as to minimize any loss of data or mail service during the swap. Currently we are busy with the webmail upgrades but it is next in our sights for projects to undertake!

[Brangwyn]

Thanks Josh, I know you guys have the hands on view of the box, but thinking out loud in the past when I've seen drive corruption occur on a machine more than once or twice I've found it's often not the drives at fault but rather CPU or RAM about to die, maybe it would be better just to do a total rebuild on new hardware and migrate the data over? that way you can rule out everything at once.

I know Marc's mentioned the webmail upgrades in passing but is there going to be an announcement made as to what's being done, when, and what impact there may be etc so we can give customers advance notice?

[JoshK]

Web mail is being worked on as we speak, this is the response I have been sending to all of the people submitting tickets concerning it. As much as I hate bulk responses in this case the volume was overwhelming.

__________________________________________________ __
Hello,

Let me first apologize for using a bulk response to everyone concerned, all of you that have spoken with me before in support know that I value personal input and would much rather fill each of your needs individually however in this case I believe I can accurately answer Most if not all of your concerns with one response.

Our current webmail server is slow. For a short time now we have noticed it is overworked and not really holding up to our expectations of it. The last few weeks it has been cumbersome and typically running under far more load than we believe will give you adequate service. We decided a long term solution was in order. However there is an obstacle to this solution, the server must stay up and in use, to many of you we realize it is "mission critical" in your businesses and hence can not be taken down even for a day for upgrades. We came up with a solution, we built a second server. This may sound easy but understand that it can not simply be placed in and the load split. the servers rely on configuration and setting that number into the millions. Every email box has settings, addresses, contact books, and many details saved to it. Our solution to this problem is as follows. The new server is complete, and being installed as I write this. We are right now, upgrading software and backing up the entire machine that is currently in place. Once this is completed, we will be moving a "copy" of the information onto the new machine, bug testing it and verifying it's function. Once up and working 100% we will then be able to take the current web mail server down for a day or two to go through it extensively and upgrade it. During this time, with the added load you and your customers may experience sluggish performance. Depending on your browser settings it may time out, or error out in some cases. There is no alternative to this. We must go through this slow period in order to accomplish our goal of putting a new server in place. I am sorry this may seem horribly inconvenient, but there really is no other feasible option at this time. HOWEVER there is another solution at your fingertips. Normal pop3 email does NOT rely on webmail at all. The problems with webmail are isolated to logging into webmail not to the individual email accounts which are serviced by our traditional mail servers. I know some of your customers may be reluctant for whatever reason to use a traditional mail client, but if you can encourage this practice for the next few days it will greatly reduce the webmail load and speed up the process of us getting it moved to a new system. I am sorry there will be some delays during this period however I believe that sometimes it's worth small delays in order to have a better functioning system in the end.


Thank you for your patience during the upgrade period, please assure your clients the traditional email system is working fine only the WEBMAIL server is affected durring this process.

If this letter does not fully answer your questions or concerns with @mail or email in general please respond back and I will do my best to field all of the questions tonight!

________________________________________________

There are multiple drives in this box, there are multiple partitions per drive. we have checked many components, regardless of configuration there is only ONE drive and it's singular partition that repeatedly sees problems. We are targeting it next.

part of the problem with a mail server unlike say a home computer, is that it is mission critical, and it's settings number in the hundreds of thousands to possibly millions of files and settings that must be configured. A terminal buisness machine or home computer I can simply install to set a hand full of end user details and it is ready to run. making a swap in those instances is easy. A mail server is quite a challenge as it would be impossible to hand configure and hence requires data migration which equals into downtime. Minimizing down time and quantity of bugs we will have to fix after the swap is what we're currently assessing.