Network Outage 4/29/2006

[generic]

Originally Posted by TicoGrande

And regarding planned/unplanned work or maintenance....so what if this was not announced.

The idea behind announcing it is, if I know for example they were going to do router maintenance at noon, and then the entire network went down, (not just one box) for some time, I would have known what was happening and would have been able to tell people something, like "still working on the router, it was scheduled work that is just taking longer than expected, we want to make sure out network is 100% before putting things back online again, etc etc....".

Thats my idea behind announcing planned work.

The unexplained outage is what had people in the dark. Nobody was expecting any maintenance, or other issues, and then bam, no network.

And nothing wrong about discussing issues here, the only way you will ever improve is to discuss what has happened, no blaming anyone, just our thoughts to make things better in the future for everyone.

[bubba]

Brad
Are you waiting for others to vote on what solution they think is best?

You may or may not get the poll results you are looking for but in the meantime will you just create a simple emergency forum like the one we had in the past.

That will serve as a temporary solution while you gather the additional votes.

BTW I think you do a great job and continue to improve. Why the old emergency forum fell thru the cracks is not clear but I rembember having to use it a few times and it served it's purpose perfectly.
Thanks,
b

[DigitalSkyline]

Isn't it funny that all those other times you've announced things like router config changes and possible 10 minutes downtime etc... this time you fail to notify us and it all goes to hell...

Remember, what can go wrong, will go wrong. - Murphy's Law

[mresell]

I think having an offsite forum is fine. I think it is critical something is put up for good that is maintained. Also, if this wasn't planned I am confused. A tech deciding to update an ip list, tho seemingly minor, sounds planned to me. I think the point about continued communication, no exception, is really valid. It is the one exception you make that will come back to haunt you. How many times has a "small" technical issue turned into something else. That isn't aimed at VT, just any real life technical issue and applies to anyone doing technical work.

[admin]

Originally Posted by DigitalSkyline

Isn't it funny that all those other times you've announced things like router config changes and possible 10 minutes downtime etc... this time you fail to notify us and it all goes to hell...
Remember, what can go wrong, will go wrong. - Murphy's Law

Yea but we did not make this change the same day another tech made this change and it did not take from what I under stand and then caused an issue days later. We have to make route changes all the time, we just brought up Level(3) direct with no issues as an example.

Had we been aware of the issue we would for sure let everyone know ahead of time. Again this was unplanned not planned on our side at all.

[TicoGrande]

Brad wrote...

The problem with moving this forum forum.vortechhosting.com as the name servers for all vortechhosting.com is here.

Well there is a simple way of doing this that preserves the Vortech domain, files, is easy, quick and totally painless.

Simply point an A record to a subdomain where you will place the forums i.e. http://forums.vortech.com
That way, you can leave all other Vortech domain data intact and just have the forums.vortech.com hosted elsewhere!

WE do this for at least 5 clients who want their main website on our Vortech server but want the subdomain on another server in a totally different geographic location. In two cases, it is exactly for this same purpose to have an offsite location. It is easy, cost free, and can be set up in about two minutes. (It is also a GREAT and profitable add-on service for you resellers!)

You also can have email for the subdomain if you want.

If you have questions on how I do it, contact me off-forum, but I have no doubt your tech experts at Vortech can do this in about two secs...

TG

[Brangwyn]

Simply point an A record to a subdomain where you will place the forums i.e. http://forums.vortech.com
That way, you can leave all other Vortech domain data intact and just have the forums.vortech.com hosted elsewhere!

I think you missed the point there, the nameservers are still in Vortech's network .. network down .. can't lookup the host record for forums.vortech.com

What needs to be done, is add manually an extra 2 or 3 external SOA Nameservers for vortechhosting.com and have those offsite at simplyDNS or somewhere so that if primary, secondary, tertiary or whatever nameservers can't be contacted due to a network outage, you should still be able to resolve the host address for the forum.

Someone should actually put onto the HSphere wishlist the ability for resellers to define additional NS servers for their service domains as well, I would dearly love to be able to mirror my service domain DNS outside of vortechs network, but ya just can't fricking well do it with HSphere currently (well you can, but it breaks RFC if you add extra nameservers at the root dns level and they're not actually listed as NS records on the SOA).

[TicoGrande]

I think you missed the point there, the nameservers are still in Vortech's network .. network down .. can't lookup the host record for forums.vortech.com

Nope... Actually... I did not miss the point.

The entire process uses a remote nameserver solely for the subdomain and NOT for the vortech.com domain (though quite frankly it would be a HUGE advantage to Vortech to use a remote nameserver for the entire Vortech.com domain as it would provide them a much needed DNS backup that would be safe and secure and allow them huge flexibilty to redirect sites and services at will, and add 3 additional nameservers in geographically different locations, etc... but I digress.

In any case, it has no effect whatsoever on the existing nameserver nor on the existing domains! Nothing at ALL need be changed if they do not wish to.

It is simply, quick and painless.... and best of all... it works!

Now saying this, there MAY be some weird Hsphere thing that could be a problem, though I doubt that very much as Hsphere uses hsphere.cc and not the vortech.com domain.

This system works perfectly for us, and I am 99.9% sure it would work for them as well.

TG

[Brangwyn]

Well, ok you didn't miss the point, but you didn't fully explain what you were trying to do. In fact having a subdomains DNS outside the SOA for the TLD is quite a bad thing to do. It invariably causes all world of problems becuase as far as I know that's not RFC compliant, I'm not saying it doesn't work (most of the time) .. but I've also seen the problems it can cause first hand with a customer who tried to transfer a subdomain setup just that way.

[TicoGrande]

Yeah, you're right... I wasn't clear in my first post, but I really didn't want to go into details.

As for being a bad idea... well, I SORTA agree with you in principle though it does work fine.

As I stated (and you implied also) the best solution is to take the DNS outside network for ALL the Vortech and Matrixreseller domains. That solves the SOA objections and it also solves a number of other important issues and adds a LOT of flexibility for Vortech to provide fast, accurate information on a timely basis.

TG

[PENorwood]

Well I for one am ok with what happened ... in four years time I have never seen an outage like this. Once or twice a decade is a pretty good record if you ask me.

[bubba]

Brad
What is the update on the emergency forum? Was a decision made to set it up?
If so please let us know as soon as it is activated.

Thanks,
b

[Dean]

The external monitor is already setup here:

http://nagios1.hsphere.cc

If DNS is down, you need to connect to this:

http://69.10.149.233/