A 24-bay SuperServer that houses databases, VMs, and the operational centre of a company. The hard drives (disks) for this system were removed during scheduled maintenance. The removal resulted in a random arrangement of the disks into the new bay configuration, and the original order had not been written down. There were no failed disks; each was functioning properly.
This is a scenario, not a case study. It illustrates a common failure type that we address on a regular basis, and how it is approached, written from the technical side (versus as part of a single customer project). Please visit our documented case files here.
One large RAID array (sixteen, twenty-four, or even more bays) has developed a fault. An individual pulls each drive out to test them separately; alternatively, they move the entire enclosure and reinsert all of the caddies into different bay locations. All of the drives are healthy. The array will not assemble.
Each disk within a striped or parity RAID is assigned a role based upon its physical position. For example, disk 3 will contain the third stripe unit of every stripe and disk 4 will contain the fourth. If you swap these two disks, each read operation will retrieve the correct bytes in the wrong sequence, so although the data exists and is complete, it is still useless.
The majority of controllers store information about their components. This is how systems can often automatically recover from an event where the drives were simply reordered. However, this information is not universally stored; it may have been damaged by the same event that caused the original fault, and may not be in a format the controller will read on software RAID configurations.
The number of possible sequences (order) of 24 devices on a storage array is extremely large. So testing every possible combination of the members will never solve the problem.
Because there is a mathematical relationship between the members of a parity array, the solution to the problem can be determined by testing candidate orders across many stripes using that relationship. When you test the candidates, the one that satisfies the equation of the parity block being equal to the XOR of the data blocks for each stripe is the correct sequence and rotation.
If you do not use parity on your striped configuration, you would need to determine what structure exists on disk. File system structures and large contiguous files both serve as references. A continuous read of a file across stripe boundaries occurs only if the members are ordered correctly.
Regardless of how the work is accomplished, the work is being done against images. All disks should be sent along with any records of which bay they were pulled from. Sending this information doesn't affect the possibility of doing the work; it affects the amount of time it will take.
Yes, almost always. A record of the original order is contained within the data itself (controller descriptor information, stripe continuity, file system landmark information and parity arithmetic). No “guessing” occurs since an order is simply read out of the data, and no failure needs to have occurred to any disk.
There are two reasons why not. There are approximately 620 billion trillion possible orders when using 24 drives. Therefore, testing is not practical. Additionally, each test may write additional controller metadata over the evidence used to reconstruct the actual order.
The four independent methods, which are cross-checked, are as follows: metadata written to each drive by the controller, data structures extending across multiple disks (e.g., stripe continuity), file system landmarks occurring at known points in a file system, and on parity arrays, whether the mathematical operations resolve. If an order is incorrect, then a failure is evident immediately.
Creating images of all 24 drives takes multiple days before any analysis can begin, and the rebuild process uses these image files instead of the original disks. Therefore, the time frame for completing the rebuild is determined by the total number of disks in the array, not by how difficult it may be to complete the rebuild.
Number your drives before you remove them (to preserve their original sequence), and take a photo of the front of your system before disassembling it. That one photo takes 5 seconds and will document the sequence for good. In most cases we've had where the sequence got lost, it was because people broke down the box quickly.
Yes, and we do check them structurally, too. You might be able to make a clean backup of an image (.vmdk) or database, but it would still refuse to start if it was taken during a write cycle, so we verify structural integrity prior to returning it.
One photograph of your fully populated chassis permanently documents the order of all components. This is the least expensive form of insurance you could buy for your array.
Labels physically attached to individual components remain on those components even after they have been moved, tested, and transferred to someone else's possession, i.e., someone who was not there when they came out.
Identifying what controller was used significantly reduces the number of possible metadata layouts and start offsets, which shortens the time to rebuild the data.