Zap! You’re toast
by Andrew Macpherson on Apr.23, 2009, under Operations
I love XEN virtual machines, they take up so much less rack space, but I’m probably missing a trick or two in terms of backing them up.
Today’s big job is recovering from whatever hit the hosting centre at 15:15 yesterday. It knocked sideways a couple of big disks. One was on the main server, a SAS disk that appears to have recovered for now, but has been promoted up the preventative maintenance stack, and needs to be replaced in due course. This was operating with Software Raid mirroring.
The other big disk was a 1T (920G really — I wish suppliers would work in base 2 like the rest of th computer industry, and that the marketing types who set out to intentionally mislead with this sort of flummery could be censured, but it will never happen, as the advertising standards authority seems to be staffed with innumerate arts graduates). Anyway this disk is Serial ATA, and was supposed to be running under the on-motherboard hardware RAID. Well as one of the meanings for supposed is ‘fondly believed’ that’s about right.
The disk containing the mirror was largely bare. Of course the main disk is complete Toast, so we’re bringing stuff back from backup onto a spare machine. To make matters more interesting the toasted machine had 7 virtual servers on board, which makes the restore exercise that much larger. It’s been a long night
And then we have to understand what went wrong, and take steps to prevent repetition. We await feedback from the hosting centre.
April 24th, 2009 on 07:29
Powerspike perhapes?