Recovering A Lost mdadm RAIDset In A Raspberry Pi-Based RAID Mirror

My previous post went over how to replace a failed USB-attached RAIDset member in a Raspberry Pi system running mdadm. Swapping out a failed RAIDset member is pretty simple, but what if the Pi or the micro SD card in the Pi fails?

Well, good news – mdadm stores metadata on the RAIDset members so even if you lose the RAID configuration on the SD card, you can recover it by scanning the disks.

I’m going to use the same Pi 4 (2GB) and two 64GB SanDisk USB sticks that I used in the previous post. They’re set up as a two disk mirror.

To simulate an SD card failure, I killed power to the Pi without shutting it down gracefully, then removed and re-imaged the card before putting it back into the Pi and powering it up. Here’s how to get your data back.

First things first – flash the appropriate Raspbian image to a micro SD card and set up your Pi like you would normally. Don’t forget to go through raspi-config and do the sudo apt update and sudo apt upgrade thing!

Once your Pi is up and running, fully up to date, and configured the way you like it, then shut it down, reattach the USB drives, and start it up again. Once it’s booted, check dmesg to see if the Pi can see the USB drives. The first USB drive will be called sda, the second will be sdb, and so on:

In my case, I only have two disks, /dev/sda and /dev/sdb and they’re both showing up. If you’re not seeing the disks, check connections and power and fix the problem before going on.

Using cat /proc/mdstat is the easiest way to check the status of your RAID array, but if you run it now, you’ll get nothing:

That’s because mdadm hasn’t been installed yet. Install it with the following:
sudo apt install mdadm

Once it’s installed, scan the disks on or attached to the Pi and create a RAIDset with any member disks found, using
sudo mdadm --assemble --scan --verbose

Notice that it found both /dev/sda and /dev/sdb, and it added them to an array.

Check the status of the array with cat /proc/mdstat:

It’s showing as [UU], which means both members are online and working properly. The array didn’t even need to resync!

Now, check the /etc/mdadm/mdadm.conf file to see if the configuration has been automatically updated:

Under “definitions of existing MD arrays” you can see the array is showing up. Also notice that it’s still called “DEV-04:0” – DEV-04 was the name of the Pi before I wiped out and rebuilt the SD card, so chances are it has the right data.

Reboot your Pi to see if it (and the array) come up cleanly. This isn’t a necessary step but I like doing it to make sure things are working and I haven’t forgotten anything.

Run blkid to see if /dev/md0 is listed:

So far, so good. Now, edit your /etc/fstab file, adding an entry for /dev/md0 but use the UUID instead of the device name:

I’m using /mnt as the mount point but it can be anywhere you want.

Once you’ve saved the file (and if everything is working properly), you should be able to mount the drive and use it with sudo mount -a

In this case, boring is good

If there are no errors and the system doesn’t hang or do something else weird, things may be working! Run df to see if /dev/md0 is mounted:

Hopefully you’ll see something similar. Check out the mounted directory and see if your data is there:

It’s there… it’s all there!

Depending on how you had things set up, you may need to change or set some permissions, but if it mounts and you can browse to it, that’s a good sign. Reboot the Pi once more to make sure the array and mounting work as expected, and if so, then everything should be good to go!

You can also use this method to move a RAIDset from one machine to another, although you should really, REALLY back up your data before you do that.

Actually, getting a backup of your stuff periodically isn’t a bad idea either. mdadm is mature and pretty stable, but a RAIDset with redundancy is no substitute for backups!

16 thoughts on “Recovering A Lost mdadm RAIDset In A Raspberry Pi-Based RAID Mirror”

  1. When I used the UUID the cause my raspberry pi to fail to boot up. I had to use the following: /dev/md/vol1 ext4 defaults,nofail,x-systemd.device-timeout=1,noatime 0 0

    1. Hello Pete!

      That’s interesting, I don’t recall having any problem using the UUID, but it’s always possible that something may have changed. Thank you for sharing what worked for you – when I get a chance I will give it a try.

      Thanks for the comment!

  2. my md0 is inactive after i updated my pi and when running the asseble scan the drive part of the raid are busy so therefore gets skipped

    root@raspberrypi:/home/pi# cat /proc/mdstat
    Personalities :
    md0 : inactive sdc[2](S) sdd[1](S) sde[4](S) sdb[0](S)
    7813529952 blocks super 1.2

    unused devices:

    root@raspberrypi:/home/pi# sudo mdadm –assemble –scan –verbose
    mdadm: looking for devices for /dev/md/0
    mdadm: /dev/sde is busy – skipping
    mdadm: /dev/sdd is busy – skipping
    mdadm: /dev/sdb is busy – skipping
    mdadm: /dev/sdc is busy – skipping

    1. Hi Joel!

      Sorry to hear you’re having trouble. Sometimes mdadm can be a little persnickety about the state of the RAID or individual disks. If the disks are all okay, try running the following:

      sudo mdadm –stop /dev/md0

      If you don’t get an error, you should then be able to run the “mdadm –assemble” line like you have above. If you’re still having trouble, try adding a “-v” to the command to get some more information.

      I hope this helps – good luck!

  3. Does this work for other RAID levels like RAID 5 as well or just for RAID 1 / mirror? I plan to build a RAID 5 on RPI.

    1. Hi Shivanand!

      I’ve played around with RAID5 but haven’t used it for any extended period of time. The mdadm documentation says it’s supported, though, so I expect it shouldn’t be a problem. Just keep in mind that RAID5 involves more calculations than some other simpler RAID schemes do; this may use up some CPU or I/O time that could be noticeable if your system is heavily used.

      That being said, I kind of wish I’d gone to RAID5 when I set my NAS up, having that extra disk work of space would’ve been great.

  4. Hi Mark! I had exactly this scenario, the SD card died! I worked through your guide but when I tried the mdadm –assemble, it complained. Apparently raid was already runnning! Anyway, followed all the other steps and its all there.
    Many thanks.

    1. Hi Paul – I’m glad things worked out! That’s interesting about the assemble, things may have changed a bit since I last set them up. I’ll have to go through it again sometime soon.

  5. Back again. I just upgraded the Pi4 to 64bit, went through your excellent instructions and voila! All there again. Many thanks.

  6. Thank you very much for taking the time to write this up. I’m fairly confident that without this I would have lost all of my data. Clearly that reveals a problem in how I was handling the backup plan for my RAID-1, but that’s another story. Key thing here is that my Pi stopped recognizing my RAID. I’m not quite sure why, but it may have been because both drives were connected to a USB hub that became erratic upon connecting a third drive and doing a large transfer (ironically was to be a backup). Anyway, upon rebooting I could see both drives mounted as sda1 and sdb1, but md0 was gone.

    Starting with “sudo mdadm –assemble –scan –verbose,” however, I was able to follow your guide to get things back up with no data lost. I was in a panicked state, and I came very close to using mdadm in a way that, I suspect, would have created a new array and likely ruined my chances for recovering the data.

    Thank you again!

  7. Alas and alack! I went back to 32-bit on the Pi4, and the assemble worked fine. Just gone back to the 64-bit (new install) and assemble doesn’t find anything. One of the drives is making a suspicious cheeping noise.

  8. Following on above:
    sudo mdadm –assemble –scan –verbose
    mdadm: looking for devices for further assembly
    mdadm: no recogniseable superblock on /dev/md/vol1
    mdadm: no recogniseable superblock on /dev/md/raspilite:01
    mdadm: /dev/sdb2 is busy – skipping
    mdadm: /dev/sdb1 is busy – skipping
    mdadm: Cannot assemble mbr metadata on /dev/sdb
    mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
    mdadm: no RAID superblock on /dev/sda

    mdadm: Found some drive for an array that is already active: /dev/md/vol1
    mdadm: giving up.
    mdadm: looking for devices for further assembly
    mdadm: /dev/sdb2 is busy – skipping
    mdadm: /dev/sdb1 is busy – skipping
    mdadm: No arrays found in config file or automatically

    It was all working fine on 64 bit before, and 32-bit.

  9. Went back to 32-bit OS. At first no luck but a reboot later found /mnt and md0. Dunno why it didn’t work at first.

  10. Hi Mark,

    Sadly, I am back here to your excellent page, without the success I had last time. I think one of the drives has lost it’s info or something.
    cat /proc/mdstat
    Personalities : [raid1] [linear] [raid0] [raid6] [raid5] [raid4] [raid10]
    md1 : active (auto-read-only) raid1 sda1[0]
    130048 blocks super 1.2 [2/1] [U_]

    md127 : inactive sda2[1](S) sdb1[0](S)
    2929550822 blocks super 1.2

    unused devices:

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.