A 24-drive SuperServer holding databases, virtual machines and the operational core of a business. During routine maintenance the drives were pulled and put back in a different order, and nobody had written down the original sequence. Not one disk was faulty.
This is a scenario, not a case study. It describes a failure pattern we handle regularly and how it is approached, written from the technical work rather than from one client's job. Our documented case files are here.
A large array — sixteen, twenty-four or more bays — develops a fault. Somebody removes the disks to test them individually, or the chassis is moved and the caddies go back in a different order. Every disk is healthy. The array will not assemble.
On a striped or parity array, position determines meaning. Disk three holds the third stripe unit of every stripe, and disk four the fourth. Swap them and every read returns the right bytes in the wrong sequence — data that is present, complete, and unusable.
Most controllers write metadata identifying each member, which is why a simple reorder is often recoverable automatically. But that metadata is not universal, it is sometimes damaged by the same event that caused the original fault, and on software arrays it may not exist in a form the controller reads.
On a 24-bay array the number of possible orderings is astronomically large, so this is never solved by trying combinations.
On a parity array the relationship between members is mathematical. For any given stripe, the parity block equals the XOR of the data blocks — and that only holds when the members are in the correct sequence with the correct rotation. Testing candidate orders against that relationship across many stripes identifies the true sequence rather than guessing it.
On a stripe with no parity, the method is different: file system structures and large contiguous files are used as the reference. A file that runs across stripe boundaries reads continuously only when the order is right.
Either way the work is done against images. Send every disk, and if you have any record of which bay each came from, send that too — it does not change what is possible, it changes how long it takes.
Almost always, yes. The order is recorded in the data itself — controller descriptors, stripe continuity, file system landmarks and parity arithmetic all point to it. It is read out rather than guessed at, and no disk needs to have failed.
No, for two reasons. With 24 drives there are around 620 billion trillion possible orders, so testing is not feasible. And each attempt risks the controller writing new metadata over the evidence needed to determine the real order.
Four independent methods, cross-checked: metadata the controller wrote to each member, data structures that continue from one disk into the next, file system landmarks at predictable positions, and on a parity array, whether the arithmetic resolves. Wrong orders fail immediately.
Because imaging 24 drives takes several days before any analysis begins, and everything runs against the images rather than the originals. The timescale is set by the number of members, not by the difficulty of the reconstruction.
Number the drives by bay before removing them, and photograph the front of the chassis first — five seconds, and it records the order permanently. Most arrays that reach us with lost sequencing were dismantled in a hurry.
Yes, and they are checked rather than just extracted. A VMDK or a database file can copy cleanly and still refuse to start if it was captured mid-write, so structural consistency is verified before it goes back.
A single photograph of the populated chassis records the order permanently. It is the cheapest insurance available on any array.
Physical labels survive being moved, being tested, and being handled by somebody who was not there when they came out.
Knowing the controller narrows the metadata layout and start offset considerably, which shortens reconstruction.