More virtual machine (VM) losses can be attributed to snapshots than to hardware faults. Each running VM is essentially a series of links instead of one individual file. Therefore, if one link is damaged, a machine will cease operation, even though its base disk remains intact. Most commonly, the sequence begins with a snapshot, taken prior to an update, that remains active for months; as additional data is added to the snapshot, it grows until the space available for storage within the datastore becomes exhausted and everything comes to a halt at once. Almost always recoverable, but not by deleting anything first.
The consistency of the pattern is substantial enough that it is worth stating first.
A person makes a snapshot of their running virtual machine prior to upgrading it. This process is done with the intention of removing the snapshot after the upgrade has been completed successfully. However, the person forgets about the snapshot and all subsequent writes go to delta files instead of the base disk. Months pass. The delta files continue to grow and eventually consume all available space on the datastore. At that point, all of the virtual machines located on the datastore stop working at once.
When things stop working, the impulse is to delete a snapshot to free up some space. Deleting a snapshot at that point is the most damaging action you can take. When deleting or consolidating a snapshot, space needs to be available to complete the process, and since all of your space was consumed by the deltas (which is why everything stopped working), you cannot complete the process. If an attempt is made to delete a snapshot while no space is available, the attempted deletion of the snapshot will leave the chain inconsistent and therefore make it impossible to boot the affected virtual machine.
Almost all of this data is still recoverable. The base disk is usually OK and the delta chain can be rebuilt. But not after someone has deleted links in it.
Do not delete snapshots to free space. If the datastore is full then add more capacity or move a machine off it. Deleting a snapshot mid-chain on a full datastore is how a recoverable situation becomes a true loss.
The physical storage failed and the datastore went with it. This is array recovery first and virtual machine work second: layers are restored from disks upward.
The array is fine, but the datastore’s own structures have been damaged. The VMDK/VHDX files are physically present; however, the map describing where they reside has been lost.
Both storage and file system healthy, and the VM will not boot because its delta chain has been broken. By far the most common of the three.
A lot of small technology firms here use consolidation very aggressively. There is typically just one server running a domain controller, a file server, a build environment, a testing environment and an application server, because this made total sense when the firm had eight employees. If you lose your datastore then you also lose six virtual systems, and there is no fallback since the fallback was also a guest operating system on that same host.
Recovering is getting the VMDK or VHDX files from the underlying volume and repairing them as guests rather than as flat files. This distinction is the entire point of recovery; a repaired guest operates normally while an extracted file simply exists.
Virtual machine recovery is priced from £800 +VAT. Where an array failure takes the datastore, both array reconstruction and guest repair are charged for as a single quote as opposed to being invoiced separately.
Each recovered VM is test-booted here prior to returning the VM wherever possible under the licensing agreement, so what we return is a machine which boots rather than a folder containing files that may boot.
Classic snapshot problem; a delta grew until there was no space left. Do not delete a snapshot to make room: consolidation requires free space to complete and interrupting it breaks the chain. Add capacity or call us first.
In most cases this is recoverable. An interrupted merge leaves parent and child inconsistent, and usually the chain can be reassembled from surviving files. Change nothing else within the folder.
Yes, and this is the common case. The array will need to be reconstructed, the datastore file system rebuilt and then the guest disks extracted and repaired. Quoted as one job.
If no data was stored to the datastore after the issue appeared, then the VM's blocks are often recoverable. If you don't shut the host down or take the datastore out of service right away, the host will quickly reuse those blocks.
They boot. Guest disks are repaired as machines and test-booted on site as long as licensing allows. A VMDK that does exist but will not boot is a non-recovery; there can be no “recovery” of a guest which refuses to boot.
The entire set of disks for the storage pool should be sent to us. The pool cannot be imported if one or more members of its vdevs are missing. Partial sets are the most common cause of delays on these jobs. Do not try to do a forced import first.
From £800 +VAT. Please tell us which hypervisor and version, how many guests were impacted by the issue and whether snapshots were in use at the time of failure. And most importantly, have you tried consolidating the VMs yet? Because that affects how much we'll have to work with.