Retrodata RAID Recovery

We recently performed what we believe to be the world’s first successful Xsan data recovery procedure using only one of the striped RAID logical volumes totally independent of the second member of the array and the metadata server.

Our client, a London University, had an Xsan storage network that consisted of two Infortrend 16-bay storage servers, each configured as RAID 3, and then striped to RAID 30. The Metadata was held on a third Infortrend server, consisting of 24 hard drives – also in RAID 3. All were connected using fibre channel and gigabit ethernet.

The Xsan system had suffered from a power spike / outage. The Metadata server and one of the Data servers shut down cleanly. However, the second Data server’s uninterruptible power supply had unknowingly been disconnected, with the result that two drives were severely damaged by the power spike.

One drive had already failed from excessive bad sectors, but the system failed to flag this correctly and, after copying the drive to the global spare, it then marked the failed disk as a global spare. This meant that the failure of only one more drive would bring the system down – but we already had a situation with multiple drive failure.

Once we’d repaired the two failed drives and the two drives with bad sectors, we made secure backups of all the drives in that data server. Then we started the long process of rebuilding the file system so that it presented identically as it was before it failed; all logical disk and logical volume ID numbers had to match the original configuration, and this was achieved by forensically editing individual bits on the hard drives.

We then delivered the recovered array member to our client and, with some final on-site configuration, we managed to bring the entire storage network back online with absolutely no data loss whatsoever.

It’s important to bear in mind that the University’s own system suppliers and consultants had deemed the data irrecoverable, as did Infortrend Technical Support, and even Apple Xsan Technical Support.

This goes to show that, even if you have been told by the highest authority that your data is not recoverable, this is almost never the case. In fact, the statements by the above Technical Support consultants could easily have convinced the client to abandon recovery, and re-install from scratch – with complete loss of data. We were told by everyone that nothing could be done. Our expertise with low-level data recovery and forensics techniques proved them all wrong.