NT26 and NT27 up and down all day

[tkraffty]

Hi - I've been watching NT26 and NT27 go up and down all day. I know these two machines have been having stability issues for a while, but I thought they had been resolved in the past few days...

My best customers happen to be hosted on those two machines, so I'm wondering if you could post a status-update on these servers? These stability issues are literally jeopardizing a third of my business revenue.

Thanks

[Vixen]

If you are having an issue, I would recommend submitting a support ticket. Brad has been out of the office, in a meeting, most of the afternoon so he has not been checking the forum.

[admin]

Yea more issues with good old MS.. We fixed the new issue with them today. It seems if MS SMTP can't send email it crashes IIS, I don't mean it can't send mail to anyone, but to just one user or domain. It does not seem to handle that very well it seems and is crashing IIS.

It took a little time to find this issue, once we matched up the times IIS crashed and logs we started seeing a pattern in the logs and times.

We fixed this issue by using the Smart host in MS SMTP to just foward its mail to anther system to send out ( a windows 2000 server ) that does not have this issue like MS SMTP does in windows 2003.

We also found that if the main DNS server drops or restarts IIS will crash.. We fixed this issue by just switching it to NS3 and NS4.

We also found a script we run to clean up the systems every night had a small issue on NT26 and NT27 for some odd reason.. Sharond fixed that already today so that should be taken care of as well.

So tonight will be the test to see how they go now.. Hope there is nothing else.. I think I summed up what has happend with these 2 systems... Good news is NT28 is online and is running Win2k with CF MX..

[Silverbug]

heh, maybe all this crashing and unstability is a sign brad the servers way of saying move us back to win2k

[Brangwyn]

Cool thanks Brad, probably a question you don't want to hear, but whats going to be the story on Credits for this month, I think it's safe to say those two servers have had a faily large amount of downtime last 30 days.

[admin]

Yea they have for sure had there share of issues..

Being that so far everything has been bugs in the OS its self kind of out of hands, let me think about and see what we can do.. The other issue is most users have had great uptime on all other systems.. Meaning 90% of there domains could of had 100% uptime and less on NT26 and NT27 due to MS bugs..

But let me think about it tonight and see what we can do on the issue. By policy it would be no since its not really "our" fault. But I am not big on that policy when it comes to things like this..

[Brangwyn]

No wurries.

One thing to perhaps consider though is you may have been able to reduce the downtime by just giving in and moving the content to a W2k box instead of persisting with trying to get the 2k3 boxes stable. Not saying you should have done that, but I think you may have had options to reduce the downtime that weren't taken which to me means there is some acountablity not only to MS but Vortech as well.

Perhaps I'm wrong though, maybe shifting user content wasn't strictly an option.

[admin]

The issue going back to win2k is it would cause a mass amount of downtime at once. About 4 to 7 hours to recreat all the sites on the new win2k option. Thats why I did not take that route and feel that we have about got all the win2k3 issues worked out. But it seems every time I say that something pop-ups up..

[Silverbug]

yeah which is a fair comment

[Brangwyn]

aye, tis the nature of the beast.

[nelsonke]

My 2 cents for the owner, I have been there.

For 25 years I have been a consutling project manager on serious voice PBX replacements and upgrades, up to 10,000 ports. When a new software load had glitches, we hated going backward, particularly when it also invovled a hardware change. It was always an impossible decision. How is one supposed to know that the next suggested fix won't permanently clear the problem.

On the other hand, if I were my best client, (who unfortunately by chance got put on NT26) I would switch back to R_____ H_____ where they were before. Losing my best client won't make me feel any better that all the other servers worked.

But, on the other hand, I understand your desire to continue to ride the new 2003's instead of going back to Win 2000's as several had recomended. It was a tough call. Thing is, we still don't know if those new fangled servers are going to work reliably.

I dont' know how much testing you did of the new Win2003's before going online, but maybe next time only install one, not two before you know for sure that they will work well.

Just a thought. Thanks for continuing to give us the whole picture.

[admin]

We did about 2 or 3 months testing along with the mssql systems running win2k3. All went well not one single OS issue. So we did setup NT26 it went fine it got a load on it and kept on going with no issues.. So we turned NT27 on. As soon as we did that it all seemed to go to hell from there. ODBC poped up with issues on both, then the DLL guy, then DNS issue and last the SMTP issue.. We have went back to win2k for all new systems untill we can see NT26 and NT27 fully stable maybe SP1 or with all of our fixes..

If you have a client that really needs the better uptime you can switch them now as NT28 is online and is win2k with no issues at all.
BTW is you said "I would switch back to R_____ H_____" then why are they on windows at all.. You do know we have FreeBSD that is VERY VERY stable.

[nelsonke]

Sorry,
I guess I came off sounding more critical than intended. I was trying to say that you have been in an tough situation, and it seems have been doing a great job. I just meant that my client on NT26 does not care that the other Win2000 and unix boxes are up. I care, but you surely understand it is not relevant to my client.

Thanks for the info on your pre launch testing. I probably should not have asked the question, based on the good job y'all do on everything else.

The R___ H_____ is not Red Hat but another Windows Hosting proivider. Their site was designed by our firm to run on ASP.

Please accept my apology for any unfair criticism that you or anyone else on the forum took from my previous post. I am very impressed and very pleased with the service you and your team provide.

[tkraffty]

Yup - I totally understand MS and other issues which are beyond your control. In this particular case, I was just a bit frustrated, only because myself and others are trying to follow the progress with these two specific machines and the daily drama that is MS - and I wasn't hearing anything else about it..

[admin]

tkraffty, not a problem at all. Its been no funny bitching at MS every day about something new and bugs to do with the new os.. I have been trying to keep everyone updated. But as you see one day the system is stable with no issue then the next both at the same time seem to find new issues they want to bitch about..

nelsonke, no issue here, we love to hear the users feed back. I try hard to make NT26 and NT27 as stable as they can be ASAP. I just got an update for the techs at the office and since we made a few more changes to NT27 its been stable since 6PM and NT26 has crash 2 times and checking the logs now on it to see if there is something else going on or that I can change to fix the issue..