01/20/2004: Emergency Maintanence - Router

[Brangwyn]

I would hesitate a guess (and last time I did I was 180 degrees wrong lol) .. that we're back on the cisco 3500 as this is exactly how the performance was earlier when apparantly that was being used.

[generic]

wow, even this forum and my email is slow now..

According to the man in charge, we're not using the core switch, but rather than core router outside of our network... It *shouldn't* be slow accordingly, but I have not a clue as to why it is. I'm just sitting at home trying to catch up on work I missed doing earlier today Rather difficult when I type 10WPM on the remote machine :/

I think somebody's working on the situation... Hopefully some other tech who is on duty will come and post more useful information.

[generic]

I give up. heading back to the bar...

[Brangwyn]

Thanks for popping in on your offtime and giving us some insight Matt It's alsmost as if we're in the middle of a DDoS attack indeed frustrating when your trying to get some work done.

[maytricks]

Exactly how long was it down this morning. I have been trying to access anything vortech since 5:00AM EST without luck. It just popped here at 8:13AM EST.

Are there going to be more outtages that I need to explain to others? According to what I read in other places i.e. Vortech Support (the only thing I could access) there were not supposed to be any major outtages after the initial situation was discovered and dealt with:

Our main Cisco Router is experiencing unexpected hardware failures. This is causing intermittent downtime for all sites on our network. We are currently working on stabilizing the router, so that it will remain up for the next hour while we prepare a replacement. Once the replacement is ready, the network will go down again for only a few minutes while we switch out the router. We apologize for the inconvenience.

I am thinking you need to make a better effort to relocate your support forums, contact information, and ticket centers to seperate routers, data centers, or whatever else can be done. This would maybe allow for frustrated customers to get some answers before they reach their boiling point.

[bocosoft]

We see that the servers are back up.
What was taking you so long?

Bostjan

[admin]

I apologize for the delay in updating all of you but our access at the office had been interrupted as well. Currently, we are routing through a backup system that has not proven to be very reliable. We are (and have been all night) working to at least restore access, until we can solve the routing problem long term.
CURRENTLY: All systems are back online as of a few minutes ago, but this may be intermittent until a more stable solution has been implemented. Rest assured that we are all working on multiple possible solutions just in case our first choice doesn't resolve the problem.

[vonbrocklin]

Originally Posted by bocosoft

We see that the servers are back up.
Bostjan

Some of the www servers might be back up, but I have not gotten anything from mail1 in at least 6 hours. I suppose it may have been working itermittently throughout the night, but when I went to bed 6 hours ago it was down and when I woke up just now it is still down.

I am sympathetic to Matrix and realize this issue is unrelated to all of the other email issues, but email keeps going down again and again and again, while none of my non-Matrix accounts ever seem to go down.



EDIT: now seems to be up...

[zennix]

Matrix people, please host your forums elsewhere from now on. Maybe you can arrange a trade with another provider. You host their support forums and they host yours. You would both have a way to communicate to your clients when things go to bad and we know what to tell our clients.

Also, as a credit to your staff, even though this outage has me concerned enough to look into other hosts, I know that finding the degree of care that you display for your resellers will likely be impossible to match. If I find one, it will just be to host support pages for my clients.

[bocosoft]

Originally Posted by vonbrocklin

I am sympathetic to Matrix and realize this issue is unrelated to all of the other email issues, but email keeps going down again and again and again, while none of my non-Matrix accounts ever seem to go down.

From my expirience, other accounts go down too (we get a lot of customers who were unhappy elsewere).
It's just that you don't notice that they go down. But as a reseller you/we have a lot of poeple noticing if stg. goes down (I bet noone calls you if our server is down and the chances you are doing bussines with our clients are minimal).

At least Matrix is keeping you informed of what is going on (if they can:-)). Our previous host had some hardware issues and the sites were down for two days and we couldn't reach them through e-mail, phone or fax.

Let's just make sure we (or should I say Matrix) has learned something from this outage.

With regards,
Bostjan

BTW: We have accounts on cPanel2 and cPanel3.

[bocosoft]

Originally Posted by zennix

Matrix people, please host your forums elsewhere from now on. Maybe you can arrange a trade with another provider. You host their support forums and they host yours. You would both have a way to communicate to your clients when things go to bad and we know what to tell our clients.

I aggre with zennix. Support should be on completely offsite servers.

Bostjan

[Probe]

unix9 is still down...its been that way for a few hours now

who knows what else is still down...the status page is down

all of my sites were down for most of the night...most are back up, but my biggest is still down. i know you guys are working on it, but am i going to get some kind of credit on my account or anything?

[Carly]

Unix9 is pinging & we're able to login to it, so it doesn't seem to be down. We are still on the backup router and experiencing intermittent problems. We're still working on getting this resolved!