(no subject)

[theducks]

It's oh so quiet.. sssh.... sssh!


So as some of you may have noticed, we had a catastrophic power failure at UWA today :) At about 9:30, the entire university lost power. It didn't get it back until about 11. Our UPS batteries last 30 minutes. I'll let you do the maths on that.

We walked into the datacentre, and it wasn't oh so quiet.. there was emergency lighting only, so it was very dark, except all the normally blue lights on the Dell servers were red (most only had one PSU on UPS). The Dells, the IPEX boxes, and the UPSes were wailing in pain and agony. And then, like so many suffocating rats, the wailings gradually quietened, as the UPSes ran out of juice and killed the devices, and then the UPS boxes stopped beeping as the batteries went totally flat. And the room was then rather dark and rather quiet.

At 11, the power came back on, for about 20 minutes, but we only had the red and some of the blue phase, and the three phase aircon wasn't able to run, so things were getting rather hot. We opened all the doors and a nice wind kept things coolish. And then we lost the blue phase mostly too, then shortly later, everything then went off again until 12. (Three phase power has three phase, red, white and blue, depending on what phase (+120,0,-120 degrees) they're on)

So we got some cool drinks, some cups, and about 15 of us just sat around in the datacentre waiting :) (ok, not just waiting, we also hit all the circuit breakers so we didn't start everything up again automatically, and did some recabling). And then, joy of joys, the power returned, and we gradually reset all the breakers, and once more, everything was good for UCS. Not a single platter loss! Apparently one of the VMS boxes had a 10 year old drive die, but it was in a RAID, so no worries.

All in all, an exciting morning :)

Movie from the darkness. Yes, we're laughing. It relieves stress, and quite frankly, without any serious UPS systems (diesel gensets, huge battery banks) there's really nothing we could have done :)

Comments

[ataxi.livejournal.com]

Might want to mention to UCCans that the mussel WAIX magic that lets me SSH in from work appears to have broken as a by-product of this cataclystrodoh.

[(Anonymous)]

There are Wheel members aware of this - it looks like the magic files that UCS (i.e. Grahame) provides have disappeared, breaking the scripts that work out what is free and what is not.

He's got a lot on his plate, so I don't know how soon it will be fixed.

- David 'Zanchey' Adam

[ataxi.livejournal.com]

Yeah, I have no problem with it (I don't expect commercial quality uptime from UCC *grin*) - just wanted to mention it since most wheel people tend not to notice for a while due to the fact they don't use the setup.

[(Anonymous)]

Should Work Now(tm).

- D'Z'A

[dannipenguin.livejournal.com]

I am fairly confident that the phases in three phase power are 120 degrees apart. If the phases were 90 degrees apart, we'd need to have 6 phase power, or very strange generators that were able to take huge sideways forces for half of their rotation.

You should also inform Grahame that he forgot to go to maths, which was conducted even with no power.

[theducks.livejournal.com]

You are of course right on both counts :)

[dannipenguin.livejournal.com]

Pfft, Alex just said there was nothing you could do. Two days in, and you're already setting a bad precedent ;)

things really get interesting when...

[gurubob.livejournal.com]

the battery banks are nearly dead and you find out that you only have half a tank of diesel left when you thought you had two.. ;) Instant Brown trouser job, that.

we had one breaker thatwouldn't stay set once after one spike - I had to hire a sparky to just sit there 24/7 to reset it for 4 days until I could get a change window to replace it.

thanks for the movie, the UPS beeping had me in the corner whimpering with flashbacks... "the horror, the horror.."

[alias_sqbr]

Meanwhile I sat working happily at home occasionally annoyed that I couldn't check my maths email :)

[velithya]

mmm, beeping