![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
(no subject)
Jul. 19th, 2005 03:39 pm( With apologies to Bjork )
So as some of you may have noticed, we had a catastrophic power failure at UWA today :) At about 9:30, the entire university lost power. It didn't get it back until about 11. Our UPS batteries last 30 minutes. I'll let you do the maths on that.
We walked into the datacentre, and it wasn't oh so quiet.. there was emergency lighting only, so it was very dark, except all the normally blue lights on the Dell servers were red (most only had one PSU on UPS). The Dells, the IPEX boxes, and the UPSes were wailing in pain and agony. And then, like so many suffocating rats, the wailings gradually quietened, as the UPSes ran out of juice and killed the devices, and then the UPS boxes stopped beeping as the batteries went totally flat. And the room was then rather dark and rather quiet.
At 11, the power came back on, for about 20 minutes, but we only had the red and some of the blue phase, and the three phase aircon wasn't able to run, so things were getting rather hot. We opened all the doors and a nice wind kept things coolish. And then we lost the blue phase mostly too, then shortly later, everything then went off again until 12. (Three phase power has three phase, red, white and blue, depending on what phase (+120,0,-120 degrees) they're on)
So we got some cool drinks, some cups, and about 15 of us just sat around in the datacentre waiting :) (ok, not just waiting, we also hit all the circuit breakers so we didn't start everything up again automatically, and did some recabling). And then, joy of joys, the power returned, and we gradually reset all the breakers, and once more, everything was good for UCS. Not a single platter loss! Apparently one of the VMS boxes had a 10 year old drive die, but it was in a RAID, so no worries.
All in all, an exciting morning :)
Movie from the darkness. Yes, we're laughing. It relieves stress, and quite frankly, without any serious UPS systems (diesel gensets, huge battery banks) there's really nothing we could have done :)
So as some of you may have noticed, we had a catastrophic power failure at UWA today :) At about 9:30, the entire university lost power. It didn't get it back until about 11. Our UPS batteries last 30 minutes. I'll let you do the maths on that.
We walked into the datacentre, and it wasn't oh so quiet.. there was emergency lighting only, so it was very dark, except all the normally blue lights on the Dell servers were red (most only had one PSU on UPS). The Dells, the IPEX boxes, and the UPSes were wailing in pain and agony. And then, like so many suffocating rats, the wailings gradually quietened, as the UPSes ran out of juice and killed the devices, and then the UPS boxes stopped beeping as the batteries went totally flat. And the room was then rather dark and rather quiet.
At 11, the power came back on, for about 20 minutes, but we only had the red and some of the blue phase, and the three phase aircon wasn't able to run, so things were getting rather hot. We opened all the doors and a nice wind kept things coolish. And then we lost the blue phase mostly too, then shortly later, everything then went off again until 12. (Three phase power has three phase, red, white and blue, depending on what phase (+120,0,-120 degrees) they're on)
So we got some cool drinks, some cups, and about 15 of us just sat around in the datacentre waiting :) (ok, not just waiting, we also hit all the circuit breakers so we didn't start everything up again automatically, and did some recabling). And then, joy of joys, the power returned, and we gradually reset all the breakers, and once more, everything was good for UCS. Not a single platter loss! Apparently one of the VMS boxes had a 10 year old drive die, but it was in a RAID, so no worries.
All in all, an exciting morning :)
Movie from the darkness. Yes, we're laughing. It relieves stress, and quite frankly, without any serious UPS systems (diesel gensets, huge battery banks) there's really nothing we could have done :)
