The building's mains power went out. Once the PowerEdge was turned back on, two hard disk drives were “missing” from a RAID 5 setup with five member drives, and it looked like the entire RAID array had been lost... until we found the two bad circuit boards in front of two perfectly good disks.
← All case files · from £500 + VAT
This is a scenario, not a case study. It illustrates a common failure type that we address on a regular basis, and how it is approached, written from the technical side (versus as part of a single customer project). Please visit our documented case files here.
It could be a power outage, the UPS didn't hold long enough, or there was a power supply problem inside the chassis. The server came back online and two members of the RAID 5 were offline and the volume wouldn't mount. At first glance this appears to be a complete loss of data.
Simultaneous physical failure of both hard drives at the same time is rare. Usually what happens after a power-related event is that one drive has some kind of actual failure and the other has simply been disconnected by the controller; the controller timed out waiting for it to spin up or saw that the states were different upon rebooting.
In many cases, the second drive is completely fine. That second disk drive contains all of your current data from that portion of the RAID. An image can then be made from that second disk drive and used during the reconstruction process just as if you never had an issue.
That’s why two failure reports by an array very often represent a single-failure recovery in fact.
Controllers using write-back caching report success after each write is written into memory (cache), not after the write is actually written onto the hard drive. A battery or supercapacitor supports this memory during a loss of power, allowing it to flush its contents upon restarting.
When the supporting battery fails (and a failed cache battery is one of the most overlooked server warnings), the cached writes are lost. At that time, the stripe set includes stripes whose data blocks and parity do not match, since some of the write was committed while other parts were not.
This explains why an array may successfully boot up after a power cut without losing any disk drives and yet be unable to mount the volume, and why the controller sometimes reports that the volume is inconsistent, rather than degraded.
Do not rebuild. If you have lost two disk members from your group, there will be no redundancy left to rebuild from. A forced rebuild will compute replacement data based upon parity that disagrees. The resulting volume will mount but contain corrupt data.
Not by rebuilding it. Each member of an array is initially copied (read-only), and then the array layout is determined from this copy set. The stripe size, disk order, parity rotation and offset are all derived from the same dataset, and the array is built virtually using those image sets. When there is a single unreadable member for a particular stripe, the parity is used to fill in that stripe. A forced controller rebuild results in the writing to disk(s) being performed. This action of forcing a rebuild is typically how an otherwise recoverable RAID array will become an irrecoverable RAID array.
The cost of recovering RAID arrays, NAS devices, or servers begins at £500 plus VAT. The price quoted is fixed in writing based upon our findings after completing our complimentary 48-hour diagnostics. We require a 50% down payment prior to performing any drive-level physical recovery services. The remaining amount due will be billed only once the recovery is completed successfully.
Only the disks: we cannot accept the server hardware itself. Disk bay order is important; therefore, before you remove your drives, please include a label on each disk showing its original bay number. If possible, also make sure to identify the model of controller associated with the removed disks. Do not run a rebuild operation or reinsert a previously ejected disk into the system.
All member disks are imaged (including both that were reported as “failed”). After imaging, both disks are evaluated independently: specifically which disk originally failed and which disk was simply dropped. Both metadata timestamp information and parity relationships assist us in determining which occurred first and which was simply dropped.
Reconstruction will use the removed (ejected) drive rather than a truly failing device for recovery. The actual failed device is treated as a single missing unit and is recovered via the parity data. If cached writes were lost during the failure of the array, those impacted stripes are identified and reported rather than silently rebuilt using the incorrect parity.
This distinction, between what has been recovered and what has been calculated, is the difference in trusting a volume versus not trusting a volume.