Rebuilding does not fail randomly. It will fail because there is something in the array that you can't read, and when you run it again, you'll apply the same maximum load to the same disks that are having trouble, usually with a worse result on the second attempt.
It goes through every member of the array (that didn't die) from start to finish, writing as it goes. On those disks already carrying unreadable sectors, that's both the most likely moment for a second failure and when wrong data gets committed.
If a rebuild does not begin to run, fails while running and/or continually fails, it is not usually a controller problem. It generally indicates an issue with the other (surviving) disks in the array: they are unable to deliver “clean” reads from each disk across its entire surface for a rebuild to be successful.
It's helpful to know this before you try again. Every time you fail to build back your data, it will take another complete pass over all of the drives (the same drives that have already proven themselves incapable of handling one), which will eventually turn a failing array into a dead array by continuing to beat up the remaining drives.
Do not start a rebuild again if it has already failed. Your question was answered when you were told it failed in the first place.
Remove the array from use, and image each individual member separately. An imaging operation performs a read of each individual drive just once, gently, in a way that allows for reading past or around problematic areas (not against them), versus how a rebuild would effectively “hammer” away at them.
After all members exist as images, the array is then recreated using those images. No additional risk is taken because there are no additional operations being performed on the existing disks; also, failed reads can now be retried on a selective basis, and the disks that survived thus far will never be used again.
However, where the business genuinely cannot afford to wait, that is a legitimate reason to attempt another rebuild. Just recognise this should be a decision, and therefore should not be done based solely upon habit. Know that a second failure during the rebuild is probable rather than merely possible.
At that point, one or more of the surviving members of your array have unreadable sectors. Every subsequent retry will require another full surface pass through drives that are already struggling. Stop and image instead.
Most controllers will allow this; however, it is rarely wise. Forcing continues after an error and could potentially write incorrect data to the new disk. The problem would be compounded (not resolved) by continuing with the process.
The new disk was written with incorrect data from a failing drive. As such, the original drives have more accurate information; therefore, do not wipe the original drives. Instead, ship all the original disks (including the one you replaced).
No. You lose meaningful information about the current state of the RAID array if you make any changes during the incident. Turn off power on the array as it is now.
It is an immediate concern because allowing this situation to continue powered up increases risk, but it is also not an urgent concern as long as the array remains powered down; it can remain powered down indefinitely without risk. Switch it off, then take your time.
From £500 +VAT after the free diagnostic, fixed in writing.