My previous post went over how to replace a failed USB-attached RAIDset member in a Raspberry Pi system running mdadm. Swapping out a failed RAIDset member is pretty simple, but what if the Pi or the micro SD card in the Pi fails?
Well, good news – mdadm stores metadata on the RAIDset members so even if you lose the RAID configuration on the SD card, you can recover it by scanning the disks.
I’m going to use the same Pi 4 (2GB) and two 64GB SanDisk USB sticks that I used in the previous post. They’re set up as a two disk mirror.
To simulate an SD card failure, I killed power to the Pi without shutting it down gracefully, then removed and re-imaged the card before putting it back into the Pi and powering it up. Here’s how to get your data back.
First things first – flash the appropriate Raspbian image to a micro SD card and set up your Pi like you would normally. Don’t forget to go through raspi-config and do the sudo apt update and sudo apt upgrade thing!
Once your Pi is up and running, fully up to date, and configured the way you like it, then shut it down, reattach the USB drives, and start it up again. Once it’s booted, check dmesg to see if the Pi can see the USB drives. The first USB drive will be called sda, the second will be sdb, and so on:
In my case, I only have two disks, /dev/sda and /dev/sdb and they’re both showing up. If you’re not seeing the disks, check connections and power and fix the problem before going on.
Using cat /proc/mdstat is the easiest way to check the status of your RAID array, but if you run it now, you’ll get nothing:
That’s because mdadm hasn’t been installed yet. Install it with the following:sudo apt install mdadm
Once it’s installed, scan the disks on or attached to the Pi and create a RAIDset with any member disks found, usingsudo mdadm --assemble --scan --verbose
Notice that it found both /dev/sda and /dev/sdb, and it added them to an array.
Check the status of the array with cat /proc/mdstat:
It’s showing as [UU], which means both members are online and working properly. The array didn’t even need to resync!
Now, check the /etc/mdadm/mdadm.conf file to see if the configuration has been automatically updated:
Under “definitions of existing MD arrays” you can see the array is showing up. Also notice that it’s still called “DEV-04:0” – DEV-04 was the name of the Pi before I wiped out and rebuilt the SD card, so chances are it has the right data.
Reboot your Pi to see if it (and the array) come up cleanly. This isn’t a necessary step but I like doing it to make sure things are working and I haven’t forgotten anything.
Run blkid to see if /dev/md0 is listed:
So far, so good. Now, edit your /etc/fstab file, adding an entry for /dev/md0 but use the UUID instead of the device name:
I’m using /mnt as the mount point but it can be anywhere you want.
Once you’ve saved the file (and if everything is working properly), you should be able to mount the drive and use it with sudo mount -a
If there are no errors and the system doesn’t hang or do something else weird, things may be working! Run df to see if /dev/md0 is mounted:
Hopefully you’ll see something similar. Check out the mounted directory and see if your data is there:
Depending on how you had things set up, you may need to change or set some permissions, but if it mounts and you can browse to it, that’s a good sign. Reboot the Pi once more to make sure the array and mounting work as expected, and if so, then everything should be good to go!
You can also use this method to move a RAIDset from one machine to another, although you should really, REALLY back up your data before you do that.
Actually, getting a backup of your stuff periodically isn’t a bad idea either. mdadm is mature and pretty stable, but a RAIDset with redundancy is no substitute for backups!
When I used the UUID the cause my raspberry pi to fail to boot up. I had to use the following: /dev/md/vol1 ext4 defaults,nofail,x-systemd.device-timeout=1,noatime 0 0
Hello Pete!
That’s interesting, I don’t recall having any problem using the UUID, but it’s always possible that something may have changed. Thank you for sharing what worked for you – when I get a chance I will give it a try.
Thanks for the comment!
my md0 is inactive after i updated my pi and when running the asseble scan the drive part of the raid are busy so therefore gets skipped
root@raspberrypi:/home/pi# cat /proc/mdstat
Personalities :
md0 : inactive sdc[2](S) sdd[1](S) sde[4](S) sdb[0](S)
7813529952 blocks super 1.2
unused devices:
root@raspberrypi:/home/pi#
root@raspberrypi:/home/pi# sudo mdadm –assemble –scan –verbose
mdadm: looking for devices for /dev/md/0
mdadm: /dev/sde is busy – skipping
mdadm: /dev/sdd is busy – skipping
mdadm: /dev/sdb is busy – skipping
mdadm: /dev/sdc is busy – skipping
Hi Joel!
Sorry to hear you’re having trouble. Sometimes mdadm can be a little persnickety about the state of the RAID or individual disks. If the disks are all okay, try running the following:
sudo mdadm –stop /dev/md0
If you don’t get an error, you should then be able to run the “mdadm –assemble” line like you have above. If you’re still having trouble, try adding a “-v” to the command to get some more information.
I hope this helps – good luck!
Does this work for RAID 5 as well or just for RAID 1 / mirror?
Does this work for other RAID levels like RAID 5 as well or just for RAID 1 / mirror? I plan to build a RAID 5 on RPI.
Hi Shivanand!
I’ve played around with RAID5 but haven’t used it for any extended period of time. The mdadm documentation says it’s supported, though, so I expect it shouldn’t be a problem. Just keep in mind that RAID5 involves more calculations than some other simpler RAID schemes do; this may use up some CPU or I/O time that could be noticeable if your system is heavily used.
That being said, I kind of wish I’d gone to RAID5 when I set my NAS up, having that extra disk work of space would’ve been great.
Hi Mark! I had exactly this scenario, the SD card died! I worked through your guide but when I tried the mdadm –assemble, it complained. Apparently raid was already runnning! Anyway, followed all the other steps and its all there.
Many thanks.
Hi Paul – I’m glad things worked out! That’s interesting about the assemble, things may have changed a bit since I last set them up. I’ll have to go through it again sometime soon.
Back again. I just upgraded the Pi4 to 64bit, went through your excellent instructions and voila! All there again. Many thanks.
Thank you very much for taking the time to write this up. I’m fairly confident that without this I would have lost all of my data. Clearly that reveals a problem in how I was handling the backup plan for my RAID-1, but that’s another story. Key thing here is that my Pi stopped recognizing my RAID. I’m not quite sure why, but it may have been because both drives were connected to a USB hub that became erratic upon connecting a third drive and doing a large transfer (ironically was to be a backup). Anyway, upon rebooting I could see both drives mounted as sda1 and sdb1, but md0 was gone.
Starting with “sudo mdadm –assemble –scan –verbose,” however, I was able to follow your guide to get things back up with no data lost. I was in a panicked state, and I came very close to using mdadm in a way that, I suspect, would have created a new array and likely ruined my chances for recovering the data.
Thank you again!
Alas and alack! I went back to 32-bit on the Pi4, and the assemble worked fine. Just gone back to the 64-bit (new install) and assemble doesn’t find anything. One of the drives is making a suspicious cheeping noise.
Following on above:
sudo mdadm –assemble –scan –verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/md/vol1
mdadm: no recogniseable superblock on /dev/md/raspilite:01
mdadm: /dev/sdb2 is busy – skipping
mdadm: /dev/sdb1 is busy – skipping
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
and
mdadm: Found some drive for an array that is already active: /dev/md/vol1
mdadm: giving up.
mdadm: looking for devices for further assembly
mdadm: /dev/sdb2 is busy – skipping
mdadm: /dev/sdb1 is busy – skipping
mdadm: No arrays found in config file or automatically
It was all working fine on 64 bit before, and 32-bit.
Went back to 32-bit OS. At first no luck but a reboot later found /mnt and md0. Dunno why it didn’t work at first.
Hi Mark,
Sadly, I am back here to your excellent page, without the success I had last time. I think one of the drives has lost it’s info or something.
cat /proc/mdstat
Personalities : [raid1] [linear] [raid0] [raid6] [raid5] [raid4] [raid10]
md1 : active (auto-read-only) raid1 sda1[0]
130048 blocks super 1.2 [2/1] [U_]
md127 : inactive sda2[1](S) sdb1[0](S)
2929550822 blocks super 1.2
unused devices:
Hi Paul,
Sorry you’re having trouble – what does the content of your /etc/mdadm/mdadm.conf file show?
Hi Mark, This si what I sent earlier.
paulr@raspi4:~ $ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : inactive sdb1[0](S) sda2[1](S)
2929550822 blocks super 1.2
md1 : active (auto-read-only) raid1 sda1[0]
130048 blocks super 1.2 [2/1] [U_]
unused devices:
paulr@raspi4:~ $ sudo mdadm –misc –detail /dev/md127
/dev/md127:
Version : 1.2
Raid Level : raid1
Total Devices : 2
Persistence : Superblock is persistent
State : inactive
Working Devices : 2
Name : raspi4:vol1 (local to host raspi4)
UUID : ccb26f1d:32e8cd66:0d72c37e:a88e9c0c
Events : 40984
Number Major Minor RaidDevice
– 8 2 – /dev/sda2
– 8 17 – /dev/sdb1
paulr@raspi4:~ $ sudo mdadm –assemble –scan –verbose /dev/md127
mdadm: looking for devices for /dev/md127
mdadm: No super block found on /dev/md/01 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/md/01
mdadm: /dev/sdb1 is busy – skipping
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sda2 is busy – skipping
mdadm: /dev/sda1 has wrong uuid.
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
paulr@raspi4:/etc/mdadm $ lsusb -d 0x1058:
Bus 002 Device 003: ID 1058:25a2 Western Digital Technologies, Inc. Elements 25A2
Bus 002 Device 002: ID 1058:2623 Western Digital Technologies, Inc. Elements SE 2623
paulr@raspi4:/etc/mdadm $
THis is mdadm.conf
paulr@raspi4:/etc/mdadm $ cat mdadm.conf
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# automatically tag new arrays as belonging to the local system
HOMEHOST
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
ARRAY /dev/md/01 metadata=1.2 UUID=25283b28:7e336c26:47c333f3:08f45f5b name=raspilite:01
ARRAY /dev/md127 metadata=1.2 UUID=ccb26f1d:32e8cd66:0d72c37e:a88e9c0c name=raspi4:vol1
# This configuration was auto-generated on Wed, 05 Jun 2024 12:05:50 +1000 by mkconf
paulr@raspi4:/etc/mdadm $
The second ARRAY line was /dev/md/vol01 (I think) . When I ran –assemble /dev/md127 before I edited it, it complained that md127 wasn’t in config.
As the UUID was the same, I just changed the device to md127 hoping that would fix it. That didn’t work.
The RAID md1 was I think a wrong attempt I made a lot earlier. It may be a partition, it’s not very big.
I tried to read the man pages but it is heavy going for someone who doesn’t know that much about it.
So it seems that the disks are OK, but it’s a matter of getting mdadm in the right mood to fix it. The
mdadm –misc –detail /dev/md127 said it was Raid1 but inactive.
Cheers
Paul
I think this is significant:
“mdadm: /dev/sda1 has wrong uuid.”
Can you run the following for each of the partitions on your RAID disk members (eg. sda1, sda2, sdb1, etc)
mdadm –misc -E /dev/sda1 (or sda2, sdb1, etc)