Replacing A Failed USB Disk In A Raspberry Pi-Based RAID Mirror

My previous post went into how to create a simple but functional NAS with a Raspberry Pi 4B and two USB-attached SATA disks. In the two weeks or so that it’s been running, the NAS I built has performed very well and has been reliable (hopefully I won’t regret typing that).

But what to do WHEN a disk fails? Disks fail – even that fancy new enterprise-grade SSD that cost an arm and a leg will fail at some point. The good news is that if you’re using mdadm to provide some kind of redundancy with your disks, things should still be working if a disk fails. The bad news is that unless you’ve got a RAIDset that can specifically tolerate more than one failure (like RAID 6), you need to replace that failed disk ASAP.

I’m confident that I’ll be able to recover from losing a disk in my shiny new NAS, but I’m not one to tempt fate so I built another RAIDset with a spare Pi and two 64GB SanDisk USB sticks to play around with instead. They’re slower than the disks so things like the speed the RAIDset syncs back up is going to be different than in my previous post.

So here’s the setup – it’s a Raspberry Pi 4B (2GB) with two 64GB USB flash drives in a RAID 0 (mirror) configuration.

Here it is, working properly, with the output of cat /proc/mdstat:

and checking to see if it’s mounted using df:

To simulate a disk failure, I removed one of the USB sticks while everything was running. Here’s the output of dmesg showing the disconnection and that mdadm is soldiering on with only one disk:

Looking at the list of USB-connected devices only shows one SanDisk device:

And now the output of cat /proc/mdstat is showing a failed disk (note the “U_”):

The good news is that yes, /dev/md0 is still mounted and usable, even though it’s in a degraded state.

I reformatted the USB stick on my Windows PC so the data that was on there was lost, then reconnected it to the Pi:

There are two SanDisk devices again.

And here’s the output of dmesg again – you can see the time difference between the failure and when the “new” disk was connected:

Note that the messages both of the failure and of the newly connected USB stick show them as sdb. It could just as easily have been sda, so make sure you check to see which one failed – and, more importantly, which one didn’t!

So now there are two disks connected again, but only one of them has the RAIDset data on it. In this case, sda is the one with the data that needs to be mirrored over. Again, it could’ve been sdb. For one last check, get the output of cat /proc/mdstat again:

Notice it says sda – that means that sda has the data we want to mirror over to the other disk, which, as the previous output of dmesg showed, is sdb.

If you are replacing a failed RAID member, the replacement must be the same size or larger than the failed member. That goes for any kind of RAID level and any type (i.e. disk mirroring or partition mirroring). Keep in mind that not all disks of the same stated capacity will actually have the same capacity, so make sure you do a bit of research before going out and spending your money on a new disk that won’t fit your current array!

Now that the disk is reconnected and showing up, copy the partition layout from the existing RAIDset disk to the new disk with the following command:

sudo sfdisk -d /dev/sdX | sudo sfdisk /dev/sdY

In this case, the existing disk is /dev/sda and the new disk is /dev/sdb:

This step isn’t needed if you’re mirroring disks (as opposed to mirroring partitions), but it’s a good idea to do it anyway – if there’s an error here, you certainly don’t want to go any further until you’ve fixed the problem.

If sfdisk worked and didn’t give you any errors, then you’re ready to add the new disk to the RAIDset with the following command:

sudo mdadm --manage /dev/md0 --add /dev/sdY

Where sdY is the new disk – in my case, sdb:

If you didn’t get any errors, run cat /proc/mdstat again and you’ll see your RAIDset is rebuilding:

Notice how it now shows that there are two active elements in md0sdb[2] and sda[0]? That’s a good sign. Keep checking every once in a while to make sure the recovery is progressing.

Once it’s done, the RAIDset should be showing as all “U” again:

If you see that, everything’s rebuilt and your RAIDset is ready to handle another disk failure.

Hopefully you never need to use this information, but if you do, I hope it helps!

13 thoughts on “Replacing A Failed USB Disk In A Raspberry Pi-Based RAID Mirror”

  1. Hey Mark! Great series on using the rpi to run the mirrored NAS. I’m going to be building this as soon as my next delivery comes along. One question though, do you have a way to get notifications of a degraded mirror set?

    1. Hi Gallo!

      Thanks for the comment! The only thing I do right now is have the Pi write the contents of /proc/mdstat to a txt file at the root of the share once a minute so I can check it to see if both disks are showing as “U”. The disk enclosures are in my field of view enough that I expect I’ll see the status light change on them.

      That being said… it’s probably a good idea to set up an email notification or something like that. I will add that to my list of projects. 🙂

      Good luck with your build!

        1. Hello – thank you for the link, that looks like it’ll do the job quite nicely. Automatically sending an email when mdadm sees a problem is a lot better than counting on me looking at the disk enclosures at the right time. Thanks for the comment!

  2. Thanks, just ran through this, a few times so if/when any of my nice new disks go byby I stand a chance of recovering the RAID:
    The smartmontools that tell me the serial number of each disk was of great help in making sure I turned off the correct “faulty” drive.
    sudo smartctl -i /dev/sdY
    also
    sudo smartctl -H /dev/sda, may be useful, to list SMART problems, if any.
    Now I need to a set up the ability to send an email if there is a detected fault.
    Thanks for all the articles.

  3. Hello,
    if I get the error “mdadm: /dev/sdb not large enough to join array” then probably the new USB-Stick is too small, correct?
    Damn -.-
    I am actually very happy with this guide, as I really have a crushed USB stick in my raid1.

    1. Hi Tom!

      Sorry it took a while to get back to you. You’ve probably already got it going by now, but you’re correct – the new USB stick is too small. I’ve run into that problem a couple of times myself – a 32GB stick from one company might be just slightly different than one from another.

      Thanks for the comment!

  4. This helped me a lot!
    In my case one of the sticks stopped working properly and somehow got stuck in an old version.
    Is there a clever way to check the raid1 for example each day is the raid1 is still working? And if not, then I will receive an E-Mail or something like that? Any ideas links or suggestions? In the www I could not find a good solution.

    1. Hello Carl!

      Sorry I didn’t get back to you sooner.

      I’m glad you found the post helpful, thanks for the comment!

      A little while ago, someone else was wondering the same question and posted a link to an article showing how to set up email so mdadm can report its status. it’s here:

    2. http://marksbench.com/electronics/replacing-a-failed-usb-disk-in-a-raspberry-pi-based-raid-mirror/#comment-1347
    3. I haven’t had a chance to try it myself but hopefully it will be useful for you.

      Good luck!

  5. Hi again, Mark,

    @raspi4:~/bin $ sudo dmesg | grep sd[ab]
    [ 2.976784] sd 0:0:0:0: [sda] 2930210816 512-byte logical blocks: (1.50 TB/1.36 TiB)
    [ 2.977628] sd 0:0:0:0: [sda] Write Protect is off
    [ 2.977646] sd 0:0:0:0: [sda] Mode Sense: 47 00 10 08
    [ 2.978422] sd 0:0:0:0: [sda] No Caching mode page found
    [ 2.978456] sd 0:0:0:0: [sda] Assuming drive cache: write through
    [ 3.144812] sd 1:0:0:0: [sdb] 3906963456 512-byte logical blocks: (2.00 TB/1.82 TiB)
    [ 3.145943] sd 1:0:0:0: [sdb] Write Protect is off
    [ 3.145965] sd 1:0:0:0: [sdb] Mode Sense: 47 00 10 08
    [ 3.153517] sd 1:0:0:0: [sdb] No Caching mode page found
    [ 3.153569] sd 1:0:0:0: [sdb] Assuming drive cache: write through
    [ 6.814304] sda: sda1
    [ 6.820634] sd 0:0:0:0: [sda] Attached SCSI disk
    [ 7.205942] sdb: sdb1 sdb2
    [ 7.248311] sd 1:0:0:0: [sdb] Attached SCSI disk
    [ 10.926186] md: kicking non-fresh sda1 from array!

    pi@raspi4:~/bin $ cat /proc/mdstat
    Personalities : [raid1]
    md126 : active (auto-read-only) raid1 sdb2[1]
    1464579520 blocks super 1.2 [2/1] [_U]
    bitmap: 5/11 pages [20KB], 65536KB chunk

    md127 : active (auto-read-only) raid1 sdb1[0]
    130048 blocks super 1.2 [2/1] [U_]

    unused devices:
    pi@raspi4:~/bin $ df -k
    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/root 14986204 8550988 5769568 60% /
    devtmpfs 781452 0 781452 0% /dev
    tmpfs 947148 0 947148 0% /dev/shm
    tmpfs 378860 1336 377524 1% /run
    tmpfs 5120 4 5116 1% /run/lock
    /dev/mmcblk0p1 258095 30521 227574 12% /boot
    tmpfs 189428 44 189384 1% /run/user/1000
    pi@raspi4:~/bin $ ls /mnt
    pi@raspi4:~/bin $

    I think md127 was a mis-made raid using an SD card, but never used.
    So what oes it mean by kicking non-fresh disk from the array?
    Should I format it and try to rebuild the array?

  6. Adding to above:

    sudo mdadm –assemble –scan –verbose
    mdadm: looking for devices for further assembly
    mdadm: no recogniseable superblock on /dev/md/vol1
    mdadm: no recogniseable superblock on /dev/md/raspilite:01
    mdadm: /dev/sdb2 is busy – skipping
    mdadm: /dev/sdb1 is busy – skipping
    mdadm: Cannot assemble mbr metadata on /dev/sdb
    mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
    mdadm: no RAID superblock on /dev/sda
    mdadm: No super block found on /dev/mmcblk0p2 (Expected magic a92b4efc, got 000000e9)
    mdadm: no RAID superblock on /dev/mmcblk0p2
    :
    :
    adm: Found some drive for an array that is already active: /dev/md/vol1
    mdadm: giving up.
    mdadm: looking for devices for further assembly
    mdadm: /dev/sdb2 is busy – skipping
    mdadm: /dev/sdb1 is busy – skipping
    mdadm: No arrays found in config file or automatically

    So it appears there’s nothing to rebuild from.

  7. Back again. After looking at /proc/mdstat, dmesg, blkid, etc, It looked like the 64-bit was seeing the disks OK. There was no entry in /etc/fstab to I followed your example and copied the UUID from blockid into /etc/fstab. Ran sudo umont -a and VOILA! /mnt was there. I don’t now how many times I have read your pages but thanks for them being there.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.