Unlike with a clicking drive, the clock on the physical hardware is a clock that has stopped. The clock that continues to run is that which represents the well-intended interventions (rebuilds, re-creations, restores), all of which write to an array with a self-description that is already unreliable.
Arithmetic varies at different RAID levels, but the sequence always remains the same: shut down the array, keep the bays in their original order, image every member read-only, then reconstruct from the copies. Depending on what RAID level you have, only the final step will vary.
Redundancy works by allowing failures to occur without making anyone aware of it. When the first disk fails, the array continues to provide access to the data just like normal: no disruption, no apparent errors, usually just an alert email no one reads or a blinking light on a chassis stored away somewhere.
Weeks or months after the first disk failed in an array, another disk will fail and the entire volume will disappear. This is the failure that users report. It explains why so many arrays have arrived here running without protection for much longer than anyone had realised.
There are practical consequences related to timing. By the time you find out that something is wrong, your margin is already gone. Also, all remaining disks are the same age as those which have already died; they were manufactured from the same manufacturing batch and used for the same amount of time.
When a rebuild occurs, the system reads every sector on each of the surviving members and writes data over the new member. If an array is in good health, this is simply normal usage. In an array that has been degraded for weeks, reading every sector from every remaining member and writing the same amount of data over the new member represents sustained hours of full-surface loading on disks which have clearly reached the end of their life.
It is exactly this level of loading that results in the second failure. An array may survive the initial failure of one of its disks but die when it attempts to do a rebuild to restore itself. These sequences represent a significant portion of all the losses we see.
If the data is more important than the downtime then image first and rebuild later. If the downtime is truly more important to your company, that is a valid business decision; just make it deliberately as opposed to by default.
Every disk, including the failed one. Especially the failed disk. On RAID 5 with two members out, the first disk to fail is frequently the more readable of the two and in many cases provides enough data to recreate the lost parity.
Bay order, written down. Take a photo of the front of the enclosure prior to removal. Writing down the bay numbers is an easy way to determine part of the array geometry. While it may take some effort to derive this information, having this information will save valuable time.
The controller, if you have it. While not required, since the configuration can be determined from the disks alone, knowing the controller make and generation will significantly shorten the work.
An honest account of what was tried. If someone has previously attempted a rebuild, removed a foreign configuration, or initialised a disk, document these actions. Documenting previous attempts at recovery will affect where we look first, and no one involved in this process will consider documenting previous failures as anything other than professional.
Not usually. Often the second disk was removed due to a timeout condition instead of failing outright. Almost all of the sectors on the second disk may contain good data. Therefore both disks need to be imaged, and then those images are used to recreate the RAID set.
Commonly happens when you've moved the enclosure or if the backplane fails. Don't accept the prompt to import or clear, as that lets the system write new metadata over the existing metadata. Send them off as-is.
This is expected and represents the majority of our effort. Disk ordering, stripe size, parity rotation and start offset are all determined by finding the arrangement where the parity maths lines up.
Absolutely not. This will be the greatest risk possible on a degraded array. Turn off power and make your decision with the drives safely powered down.
Anywhere from 5-10 working days. Each individual member of the array is copied separately prior to reconstruction, and because each of those members can take some time to image depending on their size, on larger newer disks most of the total time spent is going to be copying the members.
Prices are from £500 +VAT, regardless of the RAID level, how many members are in the array or what type of controller is used, and are fixed by us in writing after we have conducted our free 48-hour diagnostic.