Recovering A Lost mdadm RAIDset In A Raspberry Pi-Based RAID Mirror

My previous post went over how to replace a failed USB-attached RAIDset member in a Raspberry Pi system running mdadm. Swapping out a failed RAIDset member is pretty simple, but what if the Pi or the micro SD card in the Pi fails?

Well, good news – mdadm stores metadata on the RAIDset members so even if you lose the RAID configuration on the SD card, you can recover it by scanning the disks.

I’m going to use the same Pi 4 (2GB) and two 64GB SanDisk USB sticks that I used in the previous post. They’re set up as a two disk mirror.

To simulate an SD card failure, I killed power to the Pi without shutting it down gracefully, then removed and re-imaged the card before putting it back into the Pi and powering it up. Here’s how to get your data back.

First things first – flash the appropriate Raspbian image to a micro SD card and set up your Pi like you would normally. Don’t forget to go through raspi-config and do the sudo apt update and sudo apt upgrade thing!

Once your Pi is up and running, fully up to date, and configured the way you like it, then shut it down, reattach the USB drives, and start it up again. Once it’s booted, check dmesg to see if the Pi can see the USB drives. The first USB drive will be called sda, the second will be sdb, and so on:

In my case, I only have two disks, /dev/sda and /dev/sdb and they’re both showing up. If you’re not seeing the disks, check connections and power and fix the problem before going on.

Using cat /proc/mdstat is the easiest way to check the status of your RAID array, but if you run it now, you’ll get nothing:

That’s because mdadm hasn’t been installed yet. Install it with the following:
sudo apt install mdadm

Once it’s installed, scan the disks on or attached to the Pi and create a RAIDset with any member disks found, using
sudo mdadm --assemble --scan --verbose

Notice that it found both /dev/sda and /dev/sdb, and it added them to an array.

Check the status of the array with cat /proc/mdstat:

It’s showing as [UU], which means both members are online and working properly. The array didn’t even need to resync!

Now, check the /etc/mdadm/mdadm.conf file to see if the configuration has been automatically updated:

Under “definitions of existing MD arrays” you can see the array is showing up. Also notice that it’s still called “DEV-04:0” – DEV-04 was the name of the Pi before I wiped out and rebuilt the SD card, so chances are it has the right data.

Reboot your Pi to see if it (and the array) come up cleanly. This isn’t a necessary step but I like doing it to make sure things are working and I haven’t forgotten anything.

Run blkid to see if /dev/md0 is listed:

So far, so good. Now, edit your /etc/fstab file, adding an entry for /dev/md0 but use the UUID instead of the device name:

I’m using /mnt as the mount point but it can be anywhere you want.

Once you’ve saved the file (and if everything is working properly), you should be able to mount the drive and use it with sudo mount -a

In this case, boring is good

If there are no errors and the system doesn’t hang or do something else weird, things may be working! Run df to see if /dev/md0 is mounted:

Hopefully you’ll see something similar. Check out the mounted directory and see if your data is there:

It’s there… it’s all there!

Depending on how you had things set up, you may need to change or set some permissions, but if it mounts and you can browse to it, that’s a good sign. Reboot the Pi once more to make sure the array and mounting work as expected, and if so, then everything should be good to go!

You can also use this method to move a RAIDset from one machine to another, although you should really, REALLY back up your data before you do that.

Actually, getting a backup of your stuff periodically isn’t a bad idea either. mdadm is mature and pretty stable, but a RAIDset with redundancy is no substitute for backups!

Replacing A Failed USB Disk In A Raspberry Pi-Based RAID Mirror

My previous post went into how to create a simple but functional NAS with a Raspberry Pi 4B and two USB-attached SATA disks. In the two weeks or so that it’s been running, the NAS I built has performed very well and has been reliable (hopefully I won’t regret typing that).

But what to do WHEN a disk fails? Disks fail – even that fancy new enterprise-grade SSD that cost an arm and a leg will fail at some point. The good news is that if you’re using mdadm to provide some kind of redundancy with your disks, things should still be working if a disk fails. The bad news is that unless you’ve got a RAIDset that can specifically tolerate more than one failure (like RAID 6), you need to replace that failed disk ASAP.

I’m confident that I’ll be able to recover from losing a disk in my shiny new NAS, but I’m not one to tempt fate so I built another RAIDset with a spare Pi and two 64GB SanDisk USB sticks to play around with instead. They’re slower than the disks so things like the speed the RAIDset syncs back up is going to be different than in my previous post.

So here’s the setup – it’s a Raspberry Pi 4B (2GB) with two 64GB USB flash drives in a RAID 0 (mirror) configuration.

Here it is, working properly, with the output of cat /proc/mdstat:

and checking to see if it’s mounted using df:

To simulate a disk failure, I removed one of the USB sticks while everything was running. Here’s the output of dmesg showing the disconnection and that mdadm is soldiering on with only one disk:

Looking at the list of USB-connected devices only shows one SanDisk device:

And now the output of cat /proc/mdstat is showing a failed disk (note the “U_”):

The good news is that yes, /dev/md0 is still mounted and usable, even though it’s in a degraded state.

I reformatted the USB stick on my Windows PC so the data that was on there was lost, then reconnected it to the Pi:

There are two SanDisk devices again.

And here’s the output of dmesg again – you can see the time difference between the failure and when the “new” disk was connected:

Note that the messages both of the failure and of the newly connected USB stick show them as sdb. It could just as easily have been sda, so make sure you check to see which one failed – and, more importantly, which one didn’t!

So now there are two disks connected again, but only one of them has the RAIDset data on it. In this case, sda is the one with the data that needs to be mirrored over. Again, it could’ve been sdb. For one last check, get the output of cat /proc/mdstat again:

Notice it says sda – that means that sda has the data we want to mirror over to the other disk, which, as the previous output of dmesg showed, is sdb.

If you are replacing a failed RAID member, the replacement must be the same size or larger than the failed member. That goes for any kind of RAID level and any type (i.e. disk mirroring or partition mirroring). Keep in mind that not all disks of the same stated capacity will actually have the same capacity, so make sure you do a bit of research before going out and spending your money on a new disk that won’t fit your current array!

Now that the disk is reconnected and showing up, copy the partition layout from the existing RAIDset disk to the new disk with the following command:

sudo sfdisk -d /dev/sdX | sudo sfdisk /dev/sdY

In this case, the existing disk is /dev/sda and the new disk is /dev/sdb:

This step isn’t needed if you’re mirroring disks (as opposed to mirroring partitions), but it’s a good idea to do it anyway – if there’s an error here, you certainly don’t want to go any further until you’ve fixed the problem.

If sfdisk worked and didn’t give you any errors, then you’re ready to add the new disk to the RAIDset with the following command:

sudo mdadm --manage /dev/md0 --add /dev/sdY

Where sdY is the new disk – in my case, sdb:

If you didn’t get any errors, run cat /proc/mdstat again and you’ll see your RAIDset is rebuilding:

Notice how it now shows that there are two active elements in md0sdb[2] and sda[0]? That’s a good sign. Keep checking every once in a while to make sure the recovery is progressing.

Once it’s done, the RAIDset should be showing as all “U” again:

If you see that, everything’s rebuilt and your RAIDset is ready to handle another disk failure.

Hopefully you never need to use this information, but if you do, I hope it helps!

When Is A Clog Not A Clog?

My printer has been a little fussy lately. I was printing some PETG and didn’t notice any problems, but as soon as I switched to PLA the flow of the plastic out the nozzle would slow down for a bit, then go back to normal, then slow down again.

I’ve been running a 0.6mm nozzle for a while now and while it’s bigger than the stock nozzle, it can still clog, so I went through all of the usual steps – pulls at various temperatures, using the cleaning needle, trying to force the filament through by hand, but nothing seemed to fix the problem.

It was acting different than the clogs I was accustomed to, too. Normally a clog is kind of consistent – filament comes out in a spiral or diagonally or some other strange pattern. This time it was coming out nice and straight, and I didn’t notice any problems when running the extruder 10mm at a time.

Once I tried to extrude more than about 20mm, things started to act weird. Filament would come out the nozzle properly, then stop for a second or two while the extruder motor teeth turned and ground against the filament, then flow normally again. I was out of clean nozzles to try and figured that I was going to have to order another one, but the pausing had me thinking.

There are some differences between PLA and PETG. PETG usually runs at a higher temperature than PLA, but I’d even tried doing pulls with PETG to clean the nozzle and had no luck. But… I usually print PETG at about 30mm/s, while I print PLA around 50mm/s.

I thought about it some more and the next thing I came up with was that the hotend on the printer wasn’t heating enough, or was heating and cooling in cycles. It’s the same one that came with the printer, but after confirming the temperature on the display with a non-contact thermometer, I was pretty sure that wasn’t the problem.

Then I thought about the filament path from the extruder motor to the nozzle tip. On the CR-10s (and a lot of other printers), a PTFE Bowden tube connects the extruder motor assembly to the hotend. I wondered if the hotend and nozzle weren’t the problem at all. I slowly extruded filament until it was flowing out of the nozzle, then marked on the filament where it went into the extruder, then pulled it out. Then I reinserted it by hand and sure enough, there was resistance before the line on the filament had made it back to where it had been. So something was jamming it before it got to the nozzle.

I did some more pondering and then figured it out – the filament was jamming just as it was entering the hotend. As the filament heated up and melted, it went past the jam very easily, until unmelted plastic got to the same spot and jammed. A second or two later there was enough heat to melt it again, so it moved again. The reason PETG printed and PLA didn’t was because the hotend could melt the filament back far enough at the low print speed of the PETG to keep it moving, but with PLA there was too much cold filament too quickly.

A bit of swearing later (I no longer have a fingerprint on my left index finger), I had removed the Bowden tube from the hotend. This is what I found:

Melted Bowden Tube
That doesn’t look right…
Melted Bowden Tube
Ah-HA! Melted inwards. A little darkened, too.

The Bowden tube had melted inward, partially blocking the tube. I don’t know how long this has been going on, but a quick check of my notes showed that since I got the printer I’ve been running it more and more slowly, so it may have been a while.

A quick (and carefully measured) straight cut to freshen up the end and it went back into the printer… and it’s been printing beautifully since then. I haven’t tried upping the speed back to 70-80mm/s with PLA but I may do some experimenting with that soon.

Not bad for a year-old piece of plastic tubing that’s constantly exposed to temperatures above 230C. The printer came with a spare just in case, but I did some looking around and there are aftermarket tubes that are rated for higher temperatures. Might be something to look into at some point.

Long story short: if your printer is printing in a weird stop/go pattern and the extruder motor is running properly, it may be the tube, not the nozzle.

Oh, and this is as good a point as any for a reminder – if you have a 3D printer, INSTALL A SMOKE DETECTOR NEARBY. Melting plastic is generally good when printing, as long as it’s the plastic you intended to melt!

Getting Clone / 3rd Party Arduino 32U4 Boards Working On The Raspberry Pi / Linux

Ever since Ms Geek introduced me to the Arduino last April, I’ve been having a blast tinkering with them and building things. I’ve bought quite a few Arduino boards now, most of which are clones or third-party boards. A couple of them have been duds out of the package, but when I can buy four cheap boards for the cost of one “real” board, it’s tough to rationalize paying the extra money. Plus, a bit of time with a magnifier and a soldering iron fixed all but one of the boards that were DOA.

Up until two days ago, I’ve used the Arduino IDE exclusively under Windows and things have been working well. Two days ago, I installed the IDE (1.18.10) on a Raspberry Pi 4 running Raspbian Linux (Buster), and had no problems connecting to and uploading programs to MEGA328p boards like the Nano and Pro Mini.

A nice thing about Arduinos is that from a programming perspective, they’re pretty tough to permanently kill. As always, though, I make no guarantees that any of the following (or anything on this site, for that matter) will work for you. It’s worked for me so far, but it may not work for you. Who knows – it may not even work for me tomorrow. Anyway…

When I tried to do anything with a board that had the MEGA32U4 chip, though, the IDE couldn’t find it. There were no serial ports listed that pointed to the board:

No Serial in Arduino IDE
No Arduino board to be found…

The same thing happened with every single Pro Micro clone I tested (four of them from two different orders months apart) and a Keyestudio Leonardo clone. After hours of research and tinkering, I wasn’t any closer to fixing the problem, although I did disable modemmanager, which is something I discovered you should do if you want to do Arduino stuff under Linux anyway.

Just before I gave up, I extricated an authentic Leonardo from the project it was wired to and plugged it in. Immediately, a new serial port (ttyACM0) appeared in the IDE and showed a device connected to it:

Authentic Leonardo Showing Serial Port
I guess this is why buying the real deal is a good idea…

Finally, I had a starting point. Here’s an assortment of my 32U4-based Arduino boards (the Pro Micro is basically a little Leonardo):

32U4 boards
Top left: Authentic Leonardo. Top right: Clone Leonardo. Bottom: Clone Pro Micro

An easy thing to check was the 32U4 chip. They all looked pretty much the same, other than the authentic Leonardo being quite a bit older than the other boards:

Real Leonardo
Real Leonardo
Clone Leonardo
Clone Leonardo (I put the marker on there)
Pro Micro Clone
Pro Micro Clone

Since the boards all worked under Windows, I was pretty sure that despite some different components (regulators, crystals, etc) on the clone boards, all were electrically sound. So, I took a look at what the Pi said when I plugged the boards in. It turned out that Linux was seeing the clones when I plugged them in but they disconnected almost immediately:

32U4 Disconnecting
Output of dmesg after plugging in a clone board

Then I plugged the genuine Leonardo in. It showed up in dmesg and stayed connected:

Auth Leo
Output of dmesg with the authentic Leonardo

Aside from not disconnecting, the genuine Leonardo showed up a little different in dmesg. Take a look at the “New USB device found” and “New USB device strings” lines.

On the genuine Arduino: idVendor=2341, idProduct=8036, bcdDevice=1.00, Mfr=1, Product=2, SerialNumber=3

On the clones: idVendor=2341, idProduct=0036, bcdDevice=0.01, Mfr=2, Product=1, SerialNumber=0

Everything except the idVendor is different. According to the USB ID Repository, the vendor ID of 2341 belongs to Arduino. The Product ID of 8036 is associated with the Leonardo, while the Product ID of 0036 is associated with… a blank spot:

USB ID Repository
From the USB ID Repository (https://usb-ids.gowdy.us/)

I still remember going through USB hell when the original Raspberry Pi came out, so to make sure the Pi or Raspbian weren’t causing the problem, I connected the boards to an Ubuntu VM. I got the exact same results, which took the Pi off the list of suspects. I did some more research and discovered that the vendor, product, and other information presented by the MEGA32U4 isn’t hard-coded in the chip or selected by OTP fuses – it’s firmware. As in, the bootloader. Which can be replaced.

I found some instructions for replacing the bootloader on an Arduino here and used a Nano (which works fine in Linux) as the programmer. I hooked one of the 32U4-based clones to the Nano, set the IDE to use the Nano as a programmer (Tools -> Programmer -> ArduinoISP), then burned the new bootloader (Tools -> Board -> Arduino Leonardo, then Tools -> Burn Bootloader).

I hooked the clone Leonardo up to the Raspberry Pi and took a look at the output of dmesg:

32U4 after new bootloader
That looks a little more like it…

After burning the new bootloader, all of the clones showed the same values as the genuine Leonardo for idVendor, idProduct, bcdDevice, Mfr, Product, and SerialNumber… and all of the clones now show up as /dev/ttyACM0 or /dev/ttyACM1, and they are all visible and programmable in the Arduino IDE on a Pi 4 running Raspbian.

So, to sum up… if you can’t program a 32U4-based board with the Arduino IDE in Linux but it works fine in Windows, you can try the following:

  • Check dmesg and confirm that the board is connecting and disconnecting. If so, then…
  • Disable modemmanager (good idea to do this anyway), then…
  • Check dmesg and see if the board has idVendor=2341 and idProduct=0036. If so, then…
  • Buy or build an Arduino ISP. Once you’ve got it, then…
  • Hook up the board in question to the ISP and burn the correct bootloader to the board. Once that’s done, you can…
  • Check dmesg again and see if the board now has idProduct=8036. If it does, then…
  • Give it a try (don’t forget to change the IDE settings (i.e. Tools -> Programmer -> AVRISP mkII) back to program your board directly instead of using the IDE!)

Good luck!

ESP32-Cam Antenna Workaround

A while ago I got my hands on a pair of ESP32-Cam modules. They were crazy cheap and it looked like I had a great little device that would work well as a security or wildlife cam:

It’s tiny!

It was easy to program using the Arduino IDE and the example program, but no matter what I tried changing in the program it wouldn’t connect to my home wireless network.

There’s a lot of documentation out there on the ESP32 devices, but there are parts that seem to be missing. I wish I’d bookmarked the link, but I came across the comment section of one article that mentioned a 0R surface-mount resistor that selected the external antenna jack or the on-board PCB antenna. It didn’t take too long to find, and it was set to use the external antenna jack:

That explained why it wasn’t talking to the wireless – no signal. I could fix the problem in two ways: plug in an external antenna, or move the tiny little resistor so that it connected the ESP32 to the PCB antenna. I don’t have any connector that will fit that jack, so I decided to move the resistor.

**NOTE** Doing any combination or number of the following steps will definitely, POSITIVELY void the warranty of the device. It may also increase the electrical noise produced by the unit and/or its susceptibility to other sources of electrical noise. It may also cause operational or stability problems. It may also cause the device to produce wifi signals that are more powerful than allowed by law. Proceed at your own risk.

Unfortunately, between not being able to see it very well (even with a magnifier lamp), having a soldering iron with too large a tip, and having tingly fingers and shaking hands, I ended up with this:

Not only did I not manage to solder the resistor in place, I also ruined the PCB pad that connected back to the ESP32.

By this point I was getting pretty frustrated. Actually, I don’t think I’ve been that frustrated in quite a while. Fortunately, I had another ESP32-Cam that I could apply the knowledge that I gained on this one and not make the same mistake twi-bahahahaha… yeah, I ruined that one, too.

The two of them together cost me less than $15, but I hate throwing something out when it’s still “alive” (they both still booted and tried to connect to the wifi), and I was looking forward to playing around with them. Seeing as how they were pretty much garbage at this point anyway, I figured I’d see if there was anything I could do to get them working.

**NOTE** See previous note. Seriously.

I could see the PCB trace leading under the metal can, so I figured that if I was careful, I might be able to remove the can and find another soldering point to attach an antenna.

To remove the can, you don’t need a hacksaw or a Dremel… just a pair of very fine-tipped pliers or snippers. there is a small hole in the can in the inside corner right by where the antenna selection resistor pads are. Carefully grab it with the pliers/snips and pull straight upwards to make some room:

Slide the pliers in a bit more and lift up and away from the PCB antenna (towards the top of this picture). It may take a bit but the can end by the PCB antenna should break away:

Reposition and pull the can up and to the left – the right side and top should come free easily as well:

Reposition again and the last bit should break away with very little force:

The antenna connection point is – very nicely – right by where the antenna selection pads are, right between two surface-mount components with big pads and lots of solder:

So now to make the antenna. I used a piece of 26ga solid wire-wrap wire, but anything nice and thin will do 22ga is too big.

Cut a piece about 6.5cm long, then strip about 1-1.5mm and bend the stripped part at a right angle. Tin the wire, make sure it’s got a good coat of solder on it:

Bend the insulated part of the wire to whatever shape and angle helps you hold the stripped part in place at the new attachment point. Use the iron sparingly – if you’re heating and heating and it doesn’t seem to be melting, back off, give it a minute, and change your angle before trying again.

If you’re careful and lucky, you’ll end up with something like (or much better than) this:

Gently bend the antenna into the orientation you prefer, and you’re done!

Reminds me of one of those facehuggers from Alien

The next thing to try is powering it up (unless you haven’t programmed it, in which case you should probably do that now). With no can and a crappy detuned wire antenna, the ESP32-Cam was able to finally connect to my home wireless network:

FINALLY

And the camera’s video streaming works like a charm:

I suppose that instead of hanging a wire antenna off that point, I could’ve run a jumper from the new attachment point to the pad that connected to the PCB antenna (or the antenna jack, for that matter), but a) I just thought of it, and b) I don’t mind the crappy antenna look.

Fixed The Old Weather Station

I got a nice little weather station as a gift about a decade ago. It’s been pretty accurate and the display is usually nice and bright and colourful. Over the last while, however, the backlight has gone from bright, to flickering, to flashing once in a while, to completely dead. This made the unit almost useless unless it was daytime and you were standing right in front of it.

Today I cracked it open and gave fixing it a shot. It came apart pretty easily, but I learned later that with the problem I had that I didn’t need to remove the green circuit board. Removing it led to a lot of frustration and grief as it took me many tries to get everything lined up again.

Anyway, it turned out that there were two problems. One was that one of the wires wasn’t actually soldered to the backlight – there was a glob of solder and the wire was just resting against it. I thought that’d be an easy fix but unfortunately, it didn’t make any difference.

What I ended up doing was ignoring the original LED driver circuitry altogether and connected it to the power jack (tip positive). I put a two-pin female header in series to I could easily try different resistors:

1K was a little dim, 100 ohms was better, but I finally settled on 47 ohms. That made for about 110mA, well within a regular 1/4W resistor’s capacity, and considering there are four LEDs in parallel, that’s about 28mA per LED – nice and low. I’m pretty pleased with how it’s working at this point:

Nice and bright

Still, I’m going to run it for a few days to make sure that the resistor and backlight aren’t warming up before I solder the resistor in permanently and button the whole thing back up.

Hopefully this means I’ll get at least another ten years out of it.