Almost every array that arrives at our test bench lost one disk several weeks prior to someone realising there was a problem. The reason for this delay in realisation is that the redundant nature of arrays allows them to continue operating without interruption while quietly consuming the “margin” which would protect you against loss. When the volume ultimately disappears, the user typically remembers the second disk failure. This history is important: it dictates if a rebuild is even safe to initiate, and initiating a rebuild on an array that has been degraded for a month is the number one cause of data going from difficult to unrecoverable.
Knowing that helps determine what you should do next, and understanding that is the single most useful item on this page.
Redundancy operates by silently absorbing a failure. When the first disk fails, the array continues to provide access to your data just like it always did: there's no disruption, no error message anyone sees, and often nothing but an email nobody reads or a light on a chassis in a cupboard. The system remains operational with no protection.
Then weeks or months later another disk fails, and the volume disappears. This is the failure people will recall and report. That is why so many of the arrays we see here had been degraded for far longer than anyone realised prior to their arrival.
Practically speaking: when you finally find out something has failed, all your “margin” has been used up. A rebuild at this point is the highest-risk job possible, since you will be stressing all of the remaining drives across their full surface for hours, and those drives are the exact same age, same batch, same hours on them as the ones that have already failed.
When the controller is gone and there is no documentation of how it was configured, all four of these things will need to be determined using the member disks alone. That is most of the work involved in this process.
Array order and bay order may not always coincide with each other, especially after a chassis move or a hard drive swap. Determining array order involves finding parity relationships that only apply if the disks are in the correct order.
This ranges from 4 KB to 1 MB, depending on the controller and configuration being used. An incorrect value produces results that appear similar to correct results, and results that look similar to what could be correct are worse than those that clearly appear to be incorrect.
There are multiple ways of doing this, and controllers do not use one consistent method. Therefore, identifying the method used by the controller is necessary prior to attempting to rebuild any missing members of an array based upon parity.
Controllers reserve space at the beginning of each physical disk for their metadata. Each manufacturer reserves different amounts of space. For example, HP's Smart Array reserves an amount of space for its metadata, which causes generic recovery tools to fail.
The distribution corridor from Huntingdon to Milton Keynes delivers the largest and most time-sensitive array work which we perform (warehouse management and stock control systems on RAID 5 and 6). A warehouse operation unable to view its inventory will lose money hourly. These arrive as true emergencies for us and this is why we maintain enterprise SAS spares in our lab instead of ordering these when required.
The issue with the science park arrays is a different one: assemblies created by competent individuals (not storage experts) who often do not maintain records of the original stripe order or controller settings. Then from the fens we have arrays coming from process and monitoring systems, which tend to be older and usually have at least one disk that had failed for months prior to anyone noticing.
Send everything, in order. Every disk, including the one that failed first, labelled by bay number, along with the controller (if available). There is no way to reconstruct a RAID 5 that is missing two members; there is no parity to use as a base.
We quote for RAID and array recovery from £500 +VAT, irrespective of level, number of members, or type of controller. This price will also be agreed upon in writing after your complimentary 48-hour diagnostic period, and will remain constant.
All necessary mechanical repair to each member prior to assembly of the array is identified and quoted immediately upon diagnosis, rather than appearing as an additional cost at a later date.
In most cases, no. The second disk will often have only been removed from the array due to a timeout condition, rather than having physically failed completely, and its data may be almost entirely intact. Therefore both disks are imaged separately, and the array is then assembled using those images.
No. There is no greater risk than running a rebuild on an existing degraded array (hours of fully loaded surface activity on disks of the same age and batch as the one(s) that have already failed). If your data is more important to you than keeping the system up, then create an image of your data first.
That's expected, and it's most of what we do. Disk order, stripe size, parity rotation and start offset are all determined from the individual member disks, by finding the specific arrangement that will allow the parity maths to hold true.
No. Since the configuration of the array can be derived from the individual member disks, you should simply send them. Having information about the controller used (make and generation) would assist in reducing the amount of testing required to determine how the array was configured and would eliminate some of the need for trial-and-error procedures; therefore, provide this information if you have it.
Remove the disks from your server and send them to us, clearly labelled by bay location. We cannot accept the server itself, and other components such as the chassis, backplane or motherboard are rarely relevant. What is important is the disks and their original order.
Typically 5-10 working days. Once each disk is successfully imaged (read-only), the images will be used to recreate the array. Most of the time is spent imaging large modern disks, and we only begin recreating the array once all of the images have been completed.
Quoted price from £500 +VAT, once we have viewed the members. Each disk is imaged read-only and the array is rebuilt away from the controller that lost it. If the unit is still powered and offering to rebuild, switch it off prior to any further actions.