2003-08-23 unix5.hsphere.cc

Unix5 went down around 05:50 LT. I'm downtown and attempting to boot it right about now.

Service was restored around 06:45 LT.

[Brangwyn]

Thanks Alex, does it normally take 55 minutes for a unix box to reboot ? I'm sure my Windows ones don't take that long

[jetzkr8]

I don't think ANY machine takes that long to boot. But then it certainly throws some light on how long b4 a reboot is decided. I think this should be much faster.

Other factors not considered, I would go for reboot by default and cause minimal disprution rather than opting for longer fault finding exercise. If reboot doesn't solve problem, then heavy analysis can follow. this way we may avoid lengthy investigation, when a short reboot suffices.

Just my suggestion, which i consider worh considering. I dunno if you can innitiate a reboot from remote location?

Yes we can do reboots from a remote location easily, that is normally what is done FIRST, and then if things don't work the techs continue on with a fault finding "mission" and repair it, I do not know what the issue was at this time so I have no more details on that right now.

[jetzkr8]

Surely that explains the 55 minutes taken to get the machine reboot. So much for our hot-trail to fault finding. keep up the good work guys, and thanks again.

I suggest support setup a template with log of steps to be taken. This way, support just has to fill in time and comments (also serves as a checklist) so we can put these actions into perspective and not go wondering into fault-finding missions of our own. Again just a suggestion to making tech-support more transparent. It goes something like :

18:00:05 GMT fault detected
18:00:50 1-sound alarm bells
18:00:55 2-dump coffee and get on it
18:00:59 3.......5 reviving measures
19:00:59 9OKAY remote reboot
19:02:45 10Reboot not working
19:03:00 11Despatch personell ALEX to site
........so no and so on like those log thingy stuff. Maybe some handheld application to assist in logging process, post-mortem really good like this.....I am in fanatasy mode. Keep this in mind for future huh? Good going

[Brangwyn]

If reboot doesn't solve problem, then heavy analysis can follow

Actually I'm a believer that its quite important before you reboot that you do try and determine as much of the cause of the problem as possible before rebooting and potentially loosing information that could be usefull in fault finding, of course maintaining a sense of urgency whilst doing it is helpful.

Personally I'd rather have a machine take a little longer to fix properly, than have the tech have to drive back to the datacentre becuase it's fallen over again 5 minutes after they left the building which just wastes more time.

In general it takes 15-20 minutes of downtime for someone to decide that it isn't going to recover nearly as spontaneously as it crashed (no such luck), take the datacentre key off the hook and make their way downtown. You have to allow that much time for the box to recover normally from a crash and do its filesystem check and all that.

Unix 5 didn't actually go down as far as I can tell but was slowly dying until I took it down. Had to cold boot it in the end, which is pretty much the last thing we try. And physically pulling the plug is not something you can do remotely without a degree in advanced telekinetics, so a corporeal entity has to be present to do the job. We're working on that too though.

[somereseller]

You don't have remote reboot switches?

Sure we do. As I said though, in this case it needed to be booted as cold as a cold boot can be. I ended up having to physically disconnect the power supply before it would boot up all the way and start services. No sitting comfortably in the office pushing RPC buttons for this one...

[Brangwyn]

Getting out of the office and seeing sunlight wouldn't have helped your daemon complextion at all Alex!