01/20/2004: Emergency Maintanence - Router

[Vixen]

Originally Posted by logic404

I'm not sure what the equivalent US term is - it means living un-necessarily or unfairly from someone else. If you picture the guy still living at home at 35 and living off his parents, that's bludging

Edit: LIGHTBULB - "freeloading", I think is what it's called in the States!

Ok, that makes sense now. Thanks.

[Brangwyn]

Doh yeah should have exaplined what it meant I guess

Cheer up Vixen

[thecomputerpro]

Originally Posted by Brangwyn

This all sounds incredibly harsh I guess, but look at some of the top fortune 500 CEO's and consider what those people do when a supplier fails them, they don't just pat them on the back and say never mind.... sure we're not fortune 500 companies, but that doesn't change the fundamental way you should run your business.

This probably applies unless your supplier is SAP... DOH!

I think all onsite was saying was that we pay a little and do get a lot. To each his own on asking for credit or not and for how to handle thier own customers.

TCP

[zennix]

There is a seemingly never-ending supply of people who will jump at the chance to heap praise upon Matrix and defend them no matter what. There is nothing wrong with that, but it is odd in my experience! In the past I have dropped other hosts for less than this outage and I feel no qualms about doing it either.

If this board was not here, Matrix would be another faceless host and I would drop them. Routers fail, having a tested standby is pretty mandatory. Cisco has had active failover for a long time now, there is no good reason not to use it. Cisco also has incredibly good support if you pay the support contract, which pretty much every business does that relies on connectivity to stay alive.

Matrix's saving grace is the time they have put in on this forum building trust and relationships with their clients. I will heap no praise for this one, it should not have happened like it did, but I will say thanks for working on it through the night. Now please test all of the other redundancy measures

[Brangwyn]

Yes we do get value for money, I don't think anyones ever disputed that.. I'm perfectly happy with the level of service I get for my $6k per annum I contribute to the Vortech retirement fund .

Like similar business Vortech operates on a volume based cost model, think it over in your head yourself, Brads mentioned somewhere recently something like in excess of 10,000 resellers on the books @ $35 minimum a head thats not bad going .. even after wages, infrasturcture, the yacht and presents to keep Vixen happy theres probably a little left over to grab maybe a couple of spare routers.... and don't take that as a nasty comment saying I don't think Vortech aren't doing all they can to fix this problem, becuase I know based on the last 12+ months with them that they are.

SAP Yeah well I think eveyone wishes they could work for them .. but look at the horrendous fees they charge their customers for their software to be able to support the environment they have too.

[tkraffty]

Originally Posted by Vixen

Comments like that are not even necessary.

OK, Fine... I'll accept the *slap*

...but I'm still just a tad bitchy for lack of sleep and caffeine, and wondering whether or not my biggest clients are going to walk on me 'cause I look like a chump to them now...

Oh well, ...one can only hope the CEOs and Presidents of my biggest corporate clients have the same attitude as some of the posters here and will just say... "Oh that's OK... Things break... No big whoop..." -- but I don't think that's how it works in the real world...

The BSD router has been in place for the last hour and a half or so. We're keeping a close eye on it.

[thecomputerpro]

Well, aside from the chit chat, things seemed to be down again sometime during 5:00 - 5:30 MST.

My question right now is, has this particular outage been resolved totally? Or is there other things to be done and we should expect to see more downtime? (Hopefully at a predetermined time)

Thanks for the update.

TCP

(EDIT)
Thanks for the update as I was typing this post

[Brangwyn]

Thanks for the update Alex, from this side of the world anyway things seem to be running pretty smoothly.. though is feeling a little sluggish from time to time.

[webeffx]

Originally Posted by admin

Its really hard to 100% eliminate any single points of failure, but here is what we are doing a little different then before. Before we had our "main" router a cisco 7120 that did our routering just for inside the network, our core switch is a cisco 3500 this can also do routing and "should" of been able to be the back up to the router. We never tested this live because it would of taken stuff down. This was what cisco recomend to us as a good backup option if the router had an issue we could just switch over to the switch/router.


So now for what we are doing. We are setting up a openBSD router/firewall we are also building a 2nd box just as the first one that will also be a openBSD box with the same setting config and all. If the first one does have a hardware issue we will be able to switch over to the other box with little down time ( nothing like today ) and be back up and running. But there is no easy way to run 2 routers or 2 firewalls at the same time. It can be done but at some point you will end up with a single point any way. But we are trying very hard to EVER keep that from happening like today.


Just like any one else we do try to plan things the best we can but some times things just don't seem to work like they were said to or should have. Also remember that we have grow a "LOT" since this router and switch setup were done. Now we are making sure that the first one can take the load and are putting the same thing right under it the rack. So there is no question as to if it can do it or not, maybe a little better planning on our part this time. Live and learn I guess and make sure it does not happen again.. Thats what i am doing now..

Part of my company is runing ASP services. We handle alot of companies here in Houston and we have been certified by companies such as Mastercard and Chase. I will be more than happy to tell you how they attempt to eliminate the single points of failure. It can cost an arm and a leg but in the end you do not have to switch over to the backup, the gear will do it automatically.

Let me know, I will be more than happy to help.

Also I will not be requesting a refund for the down time.

Keep up the good work.

[admin]

Post is being closed.


New router is in place at this time. If you would like to talk about this issue please us the chat forum. I have clean the post up with the un-needed or un-wanted rudeness. I do understand the stress trust but do not be rude to other members here.


To end this post here is where we are right now. OpenBSD Firewall/Router is in place and working very well. Our fail over for now is the juniper M10. We will be installing and setting up a 2nd OpenBSD Firewall/Router with in the next week. We can even split the network up on 2 routers maybe if we want, but I think that may be a LOT of over kill but we will see how it goes with the one for the next week or so and make sure it can handle the load of all the IP's.

Thanks

[Brangwyn]

Thanks Brad. Yours and everyone elses efforts are appreciated.

[bigdave]

One last thing, while we were down, I did replace some CPU fans in NT21 and installed NT30 which should be online by the end of the week, I will now LOCK this thread!