A mains failure took the building down. On restart the PowerEdge reported two drives missing from a five-member RAID 5 — which sounds like the end of an array and was in fact two dead circuit boards in front of two perfectly healthy disks.
← All case files · from £500 + VAT
This is a scenario, not a case study. It describes a failure pattern we handle regularly and how it is approached, written from the technical work rather than from one client's job. Our documented case files are here.
A power cut, a UPS that did not hold, or a supply failure in the chassis. When the server comes back, two members of a RAID 5 are offline and the volume will not mount. On the face of it that is a total loss.
Genuine simultaneous mechanical failure of two drives is rare. What a power event usually produces is one drive with a real fault and one that was simply dropped by the controller — ejected for failing to respond within a timeout while it was still spinning up, or marked failed because its state was inconsistent when power returned.
That second disk is frequently in perfect condition. It contains a complete, current copy of its share of the array, and once imaged it participates in reconstruction exactly as it would have done before.
Which is why an array reporting two failures is very often a single-failure recovery in practice.
Controllers with write-back caching acknowledge writes once they are in cache, not once they are on the platters. A battery or supercapacitor holds that cache through a power loss so it can be flushed on restart.
When that battery has failed — and a failed cache battery is one of the most commonly ignored server warnings — the cached writes are lost. The array then contains a set of stripes where the data blocks and the parity disagree, because part of the write completed and part did not.
This is why an array can come back from a power cut with no failed disks at all and still refuse to mount, and why the controller sometimes reports the volume as inconsistent rather than degraded.
Do not rebuild. With two members out there is no redundancy left to rebuild from, and a forced rebuild computes replacement data from disagreeing parity. It produces a volume that mounts and contains corruption.
All members are imaged, including both that reported as failed. The two are then assessed separately: which failed first, and which was simply dropped. Metadata timestamps and the parity relationship both help establish that.
Reconstruction uses the disk that was ejected rather than genuinely failed, treating the true failure as the single missing member and recovering it from parity. Where cached writes were lost, the affected stripes are identified and reported rather than silently reconstructed from parity that is known to be wrong.
That distinction — between what was recovered and what was computed — is the difference between a volume you can trust and one you cannot.