Recovering A Lost mdadm RAIDset In A Raspberry Pi-Based RAID Mirror

My previous post went over how to replace a failed USB-attached RAIDset member in a Raspberry Pi system running mdadm. Swapping out a failed RAIDset member is pretty simple, but what if the Pi or the micro SD card in the Pi fails?

Well, good news – mdadm stores metadata on the RAIDset members so even if you lose the RAID configuration on the SD card, you can recover it by scanning the disks.

I’m going to use the same Pi 4 (2GB) and two 64GB SanDisk USB sticks that I used in the previous post. They’re set up as a two disk mirror.

To simulate an SD card failure, I killed power to the Pi without shutting it down gracefully, then removed and re-imaged the card before putting it back into the Pi and powering it up. Here’s how to get your data back.

First things first – flash the appropriate Raspbian image to a micro SD card and set up your Pi like you would normally. Don’t forget to go through raspi-config and do the sudo apt update and sudo apt upgrade thing!

Once your Pi is up and running, fully up to date, and configured the way you like it, then shut it down, reattach the USB drives, and start it up again. Once it’s booted, check dmesg to see if the Pi can see the USB drives. The first USB drive will be called sda, the second will be sdb, and so on:

In my case, I only have two disks, /dev/sda and /dev/sdb and they’re both showing up. If you’re not seeing the disks, check connections and power and fix the problem before going on.

Using cat /proc/mdstat is the easiest way to check the status of your RAID array, but if you run it now, you’ll get nothing:

That’s because mdadm hasn’t been installed yet. Install it with the following:
sudo apt install mdadm

Once it’s installed, scan the disks on or attached to the Pi and create a RAIDset with any member disks found, using
sudo mdadm --assemble --scan --verbose

Notice that it found both /dev/sda and /dev/sdb, and it added them to an array.

Check the status of the array with cat /proc/mdstat:

It’s showing as [UU], which means both members are online and working properly. The array didn’t even need to resync!

Now, check the /etc/mdadm/mdadm.conf file to see if the configuration has been automatically updated:

Under “definitions of existing MD arrays” you can see the array is showing up. Also notice that it’s still called “DEV-04:0” – DEV-04 was the name of the Pi before I wiped out and rebuilt the SD card, so chances are it has the right data.

Reboot your Pi to see if it (and the array) come up cleanly. This isn’t a necessary step but I like doing it to make sure things are working and I haven’t forgotten anything.

Run blkid to see if /dev/md0 is listed:

So far, so good. Now, edit your /etc/fstab file, adding an entry for /dev/md0 but use the UUID instead of the device name:

I’m using /mnt as the mount point but it can be anywhere you want.

Once you’ve saved the file (and if everything is working properly), you should be able to mount the drive and use it with sudo mount -a

If there are no errors and the system doesn’t hang or do something else weird, things may be working! Run df to see if /dev/md0 is mounted:

Hopefully you’ll see something similar. Check out the mounted directory and see if your data is there:

Depending on how you had things set up, you may need to change or set some permissions, but if it mounts and you can browse to it, that’s a good sign. Reboot the Pi once more to make sure the array and mounting work as expected, and if so, then everything should be good to go!

You can also use this method to move a RAIDset from one machine to another, although you should really, REALLY back up your data before you do that.

Actually, getting a backup of your stuff periodically isn’t a bad idea either. mdadm is mature and pretty stable, but a RAIDset with redundancy is no substitute for backups!