CHALLENGE ACCEPTED: ROBOT SWEEPER, PART III

Okay, so I’m making some progress. I looked around on the internet to see if I could get some ideas for what I wanted and I came up with this:

I prefer B9 from “Lost In Space” but Robby here has thumbs (and hopefully runs Python).

After some intense pencil & paper work, some printing, and some fumbling with little screws, I’ve got this:

Frankly, it’s pretty disappointing compared to the first picture but at least it’s progress.

My plan at this point is to use a Raspberry Pi Zero as the “brain”, and use an Arduino Pro Micro to handle the motor PWM because the Pi’s software PWM can be jittery. I need to figure out how to get it moving, how to control it, and how to keep itself from throwing itself down the stairs or getting jammed in a corner under the couch. Oh, and power it without an extension cord. Oh, and get it to actually sweep the floor.

No problem – only five things. Should be easy.

CHALLENGE ACCEPTED: ROBOT SWEEPER, PART II

So… last night I made a wheel. It’s not a particularly nice wheel, but it’s wheel-y and should do the job. Not a lot of progress, I’ll admit, but I have lots of time.

Or, at least I thought I did.

Just a few minutes ago I got a text from my wife – she’s at the store and they have a sale on little robot vacuums that ends in seven days.

Seven days!

CHALLENGE ACCEPTED: ROBOT SWEEPER

Spring is here and with it are beautiful sunny clear skies and lots of light. Unfortunately, due to that light and the main floor in our house being mostly hardwood (with some linoleum), every last dust mote, eyelash, and flea turd light up like little signs starting in the late afternoon.

While we’re not disgusting people, we both hate vacuuming and sweeping. It’s a pain, hauling out the vacuum is annoying and fighting with the cords and hoses isn’t fun (not like vacuuming should be, right?)

Anyway, we were just talking about this and, as neither of us volunteered to do the rest of the sweeping and/or vacuuming forever, we started to talk about buying a little robot vacuum. My wife has a friend who has a little Roomba that’s been running pretty solid for almost a decade now, and she thinks it’s the bees knees. So we went through some of the weekly flyers and took a look because vacuums were on sale, and…

GREAT NEPTUNE’S GHOST! TELL ME THAT PRICE IS A TYPO! THAT’S GOTTA BE A TYPO…

It wasn’t a typo. Those things are EXPENSIVE. Yikes.

I’m what you’d call “cheap”, so after we discussed it some more I said, “You know, sweetie, I could just build something to do that stuff, and if it fell down the stairs or over the ledge we wouldn’t be out hundreds and hundreds of dollars. Shouldn’t be too hard, and I’m pretty sure I have everything I need here already.”

She looked at me for a second and I could tell by her slightly narrowed eyes and the set of her jaw that she was thinking pretty hard. Then she said, “Okay, you can tryyyyyyy.”

As soon as she’d replied, I was already beginning to think I’d made a serious mistake, because I’d been pretty sure she was going to say “no”. In all of my electronics hobbyist and career work, 109% of my robot attempts have been utter and complete failures (and I’m pretty sure my wife knows that). But we made a deal: I have until the end of the next vacuum sale at one of the local stores to produce a working floor cleaning robot, or we’re going to buy one.

So I’m not sure what my approach is going to be. I’m not even sure what’s out there, so a bit of research might be in order. Regardless, I’d better go find some paper and a pencil. My pride and a whole bunch of money that would be better spent on potato chips and lasagna is at stake.

I’ll do my best to update my progress here so you can laugh or shake your head at my progress (or lack thereof). Who knows – I may actually build something that works…

Alright, time to get at it!

Raspberry Pi Document Scanning Host

Some years ago we bought a USB flatbed scanner. It got a ton of use, but it took up a lot of room by our computers. As time went on it spent less time hooked up to a computer and more time either in a drawer or resting on (or in) a pile of other barely-used-but-still-useful equipment.

Unfortunately, we still need to scan a lot of bills, receipts, prescription forms, insurance stuff, and all sorts of other things. Since the scanner is now, frankly, a pain in the butt to haul out, blow the dust off, and use, things tend to get left until the last minute. Of course, then there tends to be an enormous backlog of scanning to do.

I tried using a camera for a while. While it did an okay job for the most part, it really depended on lighting and patience to get good enough and consistent enough images. So the old CanoScan LiDE 700F would get put back to work.

This gave me an idea, though – what if it could just sit in a designated spot anywhere in the house and we could scan things without having to make room for it or make room for our computers? I figured that with our fancy electronic computers and the shared electronic cables and radio waves they communicate with each other over, this should be a possibility, right?

I thought about it for a bit and came up with the following requirements:

  1. It had to work without needing a desktop or laptop,
  2. It had to use a USB stick to store scans to prevent SD card wear,
  3. It had to be able to do colour, B&W, and greyscale,
  4. The DPI had to be adjustable, and
  5. There needed to be a way to easily get the images off it.

After a little bit of digging, I stumbled across Scanner Access Now Easy (SANE), and a Python package that worked with SANE – python-sane. Those two pieces of software, a Raspberry Pi, and some stitching would give me everything I needed to do what I wanted.

I’d picked up an inexpensive little touchscreen a couple of years ago that plugs directly into the Pi’s GPIO port. It’s not the fastest, it’s not the prettiest, and it’s certainly not the best, but with a 480×320 LCD with a resistive touchscreen, it’s not too shabby at all. You can find these things (and other similar devices) all over the place. The one I used is 3.5″, 480×320, and is one of the devices that uses the LCD-show drivers.

I started out with a Pi 2 I had kicking around but while CPU load and power didn’t seem to be a problem, it wasn’t able to run the scanner smoothly at higher resolutions, and the scanner would stutter so rapidly that it sounded like grinding gears.

My next attempt was going to be with a Pi 3A+ but it only has one USB port and, as I mentioned earlier, I wanted the scans to be saved to a USB stick instead of the SD card. I could’ve used a USB hub, but in my experience, fewer components means less troubleshooting.

So I ended up going with a Pi 4 B+ (2GB). It really feels like overkill, but in contrast to the Pi 2, it scanned very smoothly.

Anyway, so here’s what you’ll need:

  • A Pi 4 B+ with Raspberry Pi OS installed and updated and booting to the GUI with an account automatically logged in,
  • Your LCD drivers installed and working,
  • A SANE-compatible USB scanner,
  • A USB stick (the one I’m using is 8GB),
  • SANE installed (sudo apt install sane),
  • The python-sane module installed (sudo apt install python3-sane), and
  • Pygame installed

Once you’ve got everything installed and hooked up, start up Python from the command line:

python3

And type the following:

import sane
sane.init()
devices = sane.get_devices()
print(devices)

If, at this point, you see something like:

[('genesys:libusb:001:005', 'Canon', 'LiDE 700F', 'flatbed scanner')]

then SANE has found your scanner and you can continue. If you only see [] then there is a USB connection problem between your Pi and scanner, a power problem, or SANE doesn’t recognize your scanner. Don’t go on unless you can get SANE to find your scanner.

Next, set up your USB stick. Plug it in, remove any existing partitions on it using your favourite GUI or command line tool, then create a single ext4 partition on it.

Now, make a mount point for the USB stick. I made mine at /home/pi/scans.

Try to mount your USB stick:

sudo mount /dev/sda1 /home/pi/scans

If you don’t get an error, you should be good to go. Now edit the /etc/fstab file and add the following line at the end:

/dev/sda1 /home/pi/mnt ext4 defaults,noatime 0 0

Unmount your USB stick (sudo umount /dev/sda1), and then see if your fstab works by typing:

sudo mount -a

If you get no errors, type df and see if /dev/sda1 is listed:

If it is, you’re good to go!

Now go into the mount point you created, create a new directory called scans, and set the pi user and group as owners:

cd /home/pi/scans
mkdir scan_share
chown pi scan_share
chgrp pi scan_share

Now, set up Samba:

sudo apt install samba

Once it’s installed, edit the /etc/samba/smb.conf file. Comment out everything in the “Share Definitions” section at the end of the file and add the following:

[scans]
comment = Scanner Share
guest ok = no
browseable = yes
create mask = 0700
directory mask = 0700
read only = no
valid users = pi
path = /home/pi/scans/scan_share

Now, run the following command to give the pi user (or whichever user you want, I’m using pi) a password to use the Samba share you just created:

sudo smbpasswd -a pi

I strongly suggest not using the same password for Samba as you use for the pi account you use to log into your Pi’s console with.

Now reboot your Pi and once it comes back up, you should be able to see your Samba share on the network:

Go into “scans”, enter your username and the password you created with smbpasswd, and you should be greeted with an empty folder. Make sure you can create something in there – if not, check the permissions on your /home/pi/scans/scan_share folder.

So now you’ve confirmed your Pi can talk to your scanner, you’ve created a filesystem on a USB stick and are mounting it at boot, you’ve set up a Samba file share and have successfully logged into it and made sure you have permissions to the share. Time to glue it all together.

To do this, I used Pygame (with Python 3). I’m a huge fan of it because I’ve used it for a lot of different projects, I’m pretty comfortable with it, and there is a lot of documentation out there with no shortage of examples.

I am, however, NOT a programmer. My formative programming years were spent on a Commodore PET 4032 almost 40 years ago, and I still miss programming with line numbers. It seems to work with my setup, though.

It’s not done by any means (I could probably pass variables to a single function to set up the scan options instead of having each option set by its own function, and I need to play with the font sizes a bit more), but one of the nice things about having a site like this is that I can write down everything that’s on the 1397 pieces of scrap paper and sticky notes that I’ve been using to organize and store the information for this project and then come back to it later. So, without further ado, here’s the Python 3 code:

'''
scan-05.py

05: Partial rewrite to use functions instead of my usual giant list of nested loops

04: Now with graphical menus:
    - Tap to start
    - Select mode (Text, Grey, Colour)
        - icon sizes:
            - text: 0,125 to 155,197
            - grey: 162,125 to 315, 197
            - colour: 322,125 to 479,197
    - Select DPI (75, 150, 300, 600)
    - Select size (letter, full bed length)
    - Select Output (PNG, JPG)
    - "Another scan with these parameters?"

03: Detect touch and scan document with python-sane

02: Making progress with display graphics

User interface for document scanner (CanoScan LIDE 700F, should work for every SANE-supported USB scanner)
Uses 3.5" resistive touch display (480x320) on Pi 4 to show options/menus
Saves to appropriate folder on attached USB stick to save wear on SD card


*** GREY IS SPELLED 'GRAY' IN THE PYTHON-SANE MODULE. COLOUR IS SPELLED 'COLOR' ***

'''

import pygame, os, time, sys, sane


HEIGHT = 320
WIDTH = 480

RED = 255,0,0
GREEN = 0,255,0
BLUE = 0,0,255
YELLOW = 255,255,0
PURPLE = 255,0,255
BLACK = 0,0,0
WHITE = 255,255,255
DK_GREY = 50,50,50
MED_GREY = 127,127,127
LT_GREY = 200,200,200

BOOT_TXT = 'Starting...'
START_TXT = 'TAP TO START'
SCANNER_CHECK_TXT = 'Checking Scanner'
DPI_75_TXT = '75DPI'
DPI_150_TXT = '150DPI'
DPI_300_TXT = '300DPI'
DPI_600_TXT = '600DPI'
MODE_TEXT_TXT = ' Text '
MODE_GRAY_TXT = ' Grey '
MODE_COLOR_TXT= 'Colour'
SIZE_LETTER_TXT = '8.5x11.0'
SIZE_BED_TXT =    '8.5x11.7'
OUTPUT_PNG_TXT = 'Save as PNG (large)'
OUTPUT_JPG_TXT = 'Save as JPG (small)'
SCANNING_TXT = 'Scanning...'
SAVING_TXT = 'Saving...'
ANOTHER_TXT = 'Scan again with these settings?'
ANOTHER_YES_TXT = ' YES '
ANOTHER_NO_TXT = ' NO  '
ERR_NOSCANNER_TXT = 'NO SCANNER FOUND'
ERR_NOFILE_TXT = 'ERROR WRITING TO FILE'

go_on = True
scan_again = True

pygame.init()

pygame.display.init()

screen = pygame.display.set_mode((WIDTH, HEIGHT), pygame.FULLSCREEN)
screen.fill(BLACK)

huge_font = pygame.font.SysFont('freemono', 100)
big_font = pygame.font.SysFont('freemono', 75)
med_font = pygame.font.SysFont('freemono', 45)
small_font = pygame.font.SysFont('freemono', 25)


# *****FUNCTIONS START HERE*****
def SetMode():

    screen.fill(BLACK)
    pygame.display.flip()
    mode_text_obj = med_font.render(MODE_TEXT_TXT, True, BLACK, WHITE)
    mode_text_rect = mode_text_obj.get_rect()
    mode_text_rect.midleft = (0,160)
    screen.blit(mode_text_obj, mode_text_rect)

    mode_gray_obj = med_font.render(MODE_GRAY_TXT, True, LT_GREY, DK_GREY)
    mode_gray_rect = mode_gray_obj.get_rect()
    mode_gray_rect.center = (240,160)
    screen.blit(mode_gray_obj, mode_gray_rect)

    mode_color_obj = med_font.render(MODE_COLOR_TXT, True, RED, BLUE)
    mode_color_rect = mode_color_obj.get_rect()
    mode_color_rect.midright = (480,160)
    screen.blit(mode_color_obj, mode_color_rect)

    pygame.display.flip()
    print("E")
    pygame.event.clear()

    go_on = False


    while go_on == False:

        for event in pygame.event.get():
            if event.type == pygame.KEYDOWN:
                if event.unicode == 'z':
                    pygame.quit()
                    sys.exit
            if event.type == pygame.MOUSEBUTTONUP:
                pos = pygame.mouse.get_pos()
                pygame.event.clear()
                if (pos[0] > 0 and pos[0] < 155 and pos[1] > 125 and pos[1] < 197):
                    print("TEXT")
                    scanner.mode = 'Lineart'
                    go_on = True
                elif (pos[0] > 162 and pos[0] < 315 and pos[1] > 125 and pos[1] < 197):
                    print("GREY")
                    scanner.mode = 'Gray'
                    go_on = True
                elif (pos[0] > 322 and pos[0] < 479 and pos[1] > 125 and pos[1] < 197):
                    print("COLOUR")
                    scanner.mode = 'Color'
                    go_on = True
                else:
                    print("AREA OUTSIDE MODE BUTTONS PRESSED")
                    go_on = False

    go_on = True

    return()


def SetDPI():

    # Options are 75, 150, 300, 600, put them at top left, bottom left, top right, bottom right
    screen.fill(BLACK)
    pygame.display.flip()
    DPI_75_obj = med_font.render(DPI_75_TXT, True, WHITE, BLACK)
    DPI_75_rect = DPI_75_obj.get_rect()
    DPI_75_rect.topleft = (0,0)
    screen.blit(DPI_75_obj, DPI_75_rect)

    DPI_150_obj = med_font.render(DPI_150_TXT, True, WHITE, BLACK)
    DPI_150_rect = DPI_150_obj.get_rect()
    DPI_150_rect.bottomleft = (0,320)
    screen.blit(DPI_150_obj, DPI_150_rect)

    DPI_300_obj = med_font.render(DPI_300_TXT, True, WHITE, BLACK)
    DPI_300_rect = DPI_300_obj.get_rect()
    DPI_300_rect.topright = (480,0)
    screen.blit(DPI_300_obj, DPI_300_rect)

    DPI_600_obj = med_font.render(DPI_600_TXT, True, WHITE, BLACK)
    DPI_600_rect = DPI_600_obj.get_rect()
    DPI_600_rect.bottomright = (480,320)
    screen.blit(DPI_600_obj, DPI_600_rect)

    pygame.display.flip()
    print("E")
    pygame.event.clear()

    go_on = False

    while go_on == False:

        for event in pygame.event.get():
            if event.type == pygame.KEYDOWN:
                if event.unicode == 'z':
                    pygame.quit()
                    sys.exit
            if event.type == pygame.MOUSEBUTTONUP:
                pos = pygame.mouse.get_pos()
                pygame.event.clear()
                if (pos[0] > 0 and pos[0] < 240 and pos[1] > 0 and pos[1] < 160):
                    print("75DPI")
                    scanner.resolution = 75
                    go_on = True
                elif (pos[0] > 0 and pos[0] < 240 and pos[1] > 161 and pos[1] < 320):
                    print("150DPI")
                    scanner.resolution = 150
                    go_on = True
                elif (pos[0] > 241 and pos[0] < 479 and pos[1] > 0 and pos[1] < 160):
                    print("300DPI")
                    scanner.resolution = 300
                    go_on = True
                elif (pos[0] > 241 and pos[0] < 479 and pos[1] > 161 and pos[1] < 320):
                    print("600DPI")
                    scanner.resolution = 600
                    go_on = True

                else:
                    print("AREA OUTSIDE MODE BUTTONS PRESSED")
                    go_on = False

    time.sleep(1)
    go_on = True

    return()


def SetSize():
    # Options are Letter (8.5x11") and full bed (8.5x11.7)
    # br_x = 216.0699920654297 for both
    # br_y = 279.4 for Letter and 297 for full bed
    screen.fill(BLACK)
    pygame.display.flip()


    SIZE_LETTER_obj = med_font.render(SIZE_LETTER_TXT, True, WHITE, BLACK)
    SIZE_LETTER_rect = SIZE_LETTER_obj.get_rect()
    SIZE_LETTER_rect.midleft = (0,160)
    screen.blit(SIZE_LETTER_obj, SIZE_LETTER_rect)

    SIZE_BED_obj = med_font.render(SIZE_BED_TXT, True, WHITE, BLACK)
    SIZE_BED_rect = SIZE_BED_obj.get_rect()
    SIZE_BED_rect.midright = (480,160)
    screen.blit(SIZE_BED_obj, SIZE_BED_rect)

    pygame.display.flip()
    pygame.event.clear()

    go_on = False

    while go_on == False:

        for event in pygame.event.get():
            if event.type == pygame.KEYDOWN:
                if event.unicode == 'z':
                    pygame.quit()
                    sys.exit
            if event.type == pygame.MOUSEBUTTONUP:
                pos = pygame.mouse.get_pos()
                pygame.event.clear()
                if (pos[0] > 0 and pos[0] < 240 and pos[1] > 0 and pos[1] < 320):
                    print("8.5x11")
                    scanner.br_x = 216.0699920654297
                    scanner.br_y = 279.4
                    go_on = True
                elif (pos[0] > 240 and pos[0] < 480 and pos[1] > 0 and pos[1] < 320):
                    print("8.5x11.7")
                    scanner.br_x = 216.0699920654297
                    scanner.br_y = 297.0
                    go_on = True

                else:
                    print("AREA OUTSIDE MODE BUTTONS PRESSED")
                    go_on = False

    time.sleep(1)
    go_on = True

    return()


def SetFileType():

    screen.fill(BLACK)
    pygame.display.flip()


    OUTPUT_PNG_obj = med_font.render(OUTPUT_PNG_TXT, True, WHITE, BLACK)
    OUTPUT_PNG_rect = OUTPUT_PNG_obj.get_rect()
    OUTPUT_PNG_rect.topleft = (0,0)
    screen.blit(OUTPUT_PNG_obj, OUTPUT_PNG_rect)

    OUTPUT_JPG_obj = med_font.render(OUTPUT_JPG_TXT, True, WHITE, BLACK)
    OUTPUT_JPG_rect = OUTPUT_JPG_obj.get_rect()
    OUTPUT_JPG_rect.bottomleft = (0,320)
    screen.blit(OUTPUT_JPG_obj, OUTPUT_JPG_rect)


    pygame.display.flip()
    pygame.event.clear()

    go_on = False

    while go_on == False:

        for event in pygame.event.get():
            if event.type == pygame.KEYDOWN:
                if event.unicode == 'z':
                    pygame.quit()
                    sys.exit
            if event.type == pygame.MOUSEBUTTONUP:
                pos = pygame.mouse.get_pos()
                pygame.event.clear()
                if (pos[0] > 0 and pos[0] < 480 and pos[1] > 0 and pos[1] < 160):
                    print("PNG SCAN")
                    screen.fill(BLACK)

                    SCANNING_obj = med_font.render(SCANNING_TXT, True, WHITE, BLACK)
                    SCANNING_rect = SCANNING_obj.get_rect()
                    SCANNING_rect.center = (240,160)
                    screen.blit(SCANNING_obj, SCANNING_rect)
                    pygame.display.flip()
                    scanner.start()
                    im = scanner.snap()
                    # Now save the scanned image
                    screen.fill(BLACK)
                    SAVING_obj = med_font.render(SAVING_TXT, True, WHITE, BLACK)
                    SAVING_rect = SAVING_obj.get_rect()
                    SAVING_rect.center = (240,160)
                    screen.blit(SAVING_obj, SAVING_rect)
                    pygame.display.flip()
                    filename = '/home/pi/scans/scan_share/' + time.strftime('%Y-%m-%d_%H%M-%S') + '.png'
                    im.save(filename)
                    time.sleep(0.5)

                    go_on = True
                elif (pos[0] > 0 and pos[0] < 480 and pos[1] > 160 and pos[1] < 320):
                    print("JPG SCAN")
                    screen.fill(BLACK)

                    SCANNING_obj = med_font.render(SCANNING_TXT, True, WHITE, BLACK)
                    SCANNING_rect = SCANNING_obj.get_rect()
                    SCANNING_rect.center = (240,160)
                    screen.blit(SCANNING_obj, SCANNING_rect)
                    pygame.display.flip()
                    scanner.start()
                    im = scanner.snap()
                    # Now save the scanned image
                    screen.fill(BLACK)
                    SAVING_obj = med_font.render(SAVING_TXT, True, WHITE, BLACK)
                    SAVING_rect = SAVING_obj.get_rect()
                    SAVING_rect.center = (240,160)
                    screen.blit(SAVING_obj, SAVING_rect)
                    pygame.display.flip()
                    filename = '/home/pi/scans/scan_share/' + time.strftime('%Y-%m-%d_%H%M-%S') + '.jpg'
                    im.save(filename)
                    print(filename)

                    go_on = True

                else:
                    print("AREA OUTSIDE MODE BUTTONS PRESSED")
                    go_on = False

    time.sleep(1)
    go_on = True

    return()


# *****FUNCTIONS END HERE*****


boot_text_obj = med_font.render(BOOT_TXT, True, WHITE, BLACK)
boot_text_rect = boot_text_obj.get_rect()
boot_text_rect.center = (240, 160)
screen.blit(boot_text_obj, boot_text_rect)
pygame.display.flip()
time.sleep(2)


# Main loop here

while True:

    screen.fill(BLACK)
    start_text_obj = med_font.render('TAP TO START', True, MED_GREY, BLACK)
    start_text_rect = start_text_obj.get_rect()
    start_text_rect.center = (240, 160)

    screen.blit(start_text_obj, start_text_rect)

    pygame.display.flip()

    for event in pygame.event.get():
        if event.type == pygame.KEYDOWN:
            if event.unicode == 'z':
                pygame.quit()
                sys.exit()
        if event.type == pygame.MOUSEBUTTONDOWN:
            screen.fill(BLACK)
            scanner_check_text_obj = med_font.render(SCANNER_CHECK_TXT, True, LT_GREY, BLACK)
            scanner_check_text_rect = scanner_check_text_obj.get_rect()
            scanner_check_text_rect.center = (240,160)
            screen.blit(scanner_check_text_obj, scanner_check_text_rect)
            pygame.display.flip()
            pygame.event.clear()
            sane.init()
            devices = sane.get_devices()
            print("A")
            if (devices == []):
                go_on = False
                screen.fill(BLACK)
                err_text_obj = small_font.render('SCANNER NOT FOUND', True, RED, BLACK)
                err_text_rect = err_text_obj.get_rect()
                err_text_rect.center = (240,160)
                screen.blit(err_text_obj, err_text_rect)
                pygame.display.flip()
                time.sleep(5)
                print("B")
            else:
                go_on = True
                print("C")

            if (go_on == True):
                print("D")
                # If we're here, open the scanner ask for and set the mode ('Lineart', 'Gray', 'Color')
                scanner = sane.open(devices[0][0])
                SetMode()

                print(scanner.mode)

                # If we're here, ask for and set the DPI (75, 150, 300, 600)    
                SetDPI()
                print("E")
                print(scanner.resolution)

                # If we're here, ask for and set the scan size (Letter, Full Bed)
                SetSize()
                print("F")
                print(scanner.area)

                # If we're here, ask for and set the output format (JPG, PNG), do the scan, and
                # save the file.
                SetFileType()
                print("G")
                #print(filename)


                #If we're here, check to see if another scan should be done with same settings
                # ANOTHER_TXT, ANOTHER_YES_TXT, ANOTHER_NO_TXT
                while scan_again == True:
                    screen.fill(BLACK)
                    ANOTHER_obj = small_font.render(ANOTHER_TXT, True, WHITE, BLACK)
                    ANOTHER_rect = ANOTHER_obj.get_rect()
                    ANOTHER_rect.midtop = (240,0)
                    screen.blit(ANOTHER_obj, ANOTHER_rect)

                    ANOTHER_YES_obj = big_font.render(ANOTHER_YES_TXT, True, BLACK, WHITE)
                    ANOTHER_YES_rect = ANOTHER_YES_obj.get_rect()
                    ANOTHER_YES_rect.bottomleft = (0,320)
                    screen.blit(ANOTHER_YES_obj, ANOTHER_YES_rect)

                    ANOTHER_NO_obj = big_font.render(ANOTHER_NO_TXT, True, BLACK, WHITE)
                    ANOTHER_NO_rect = ANOTHER_NO_obj.get_rect()
                    ANOTHER_NO_rect.bottomright = (480,320)
                    screen.blit(ANOTHER_NO_obj, ANOTHER_NO_rect)

                    pygame.display.flip()

                    #pygame.event.clear()

                    for event in pygame.event.get():
                        if event.type == pygame.KEYDOWN:
                            if event.unicode == 'z':
                                print("H")
                                pygame.quit()
                                sys.exit
                        if event.type == pygame.MOUSEBUTTONUP:
                            print("I")
                            pos = pygame.mouse.get_pos()
                            pygame.event.clear()
                            if (pos[0] > 0 and pos[0] < 240 and pos[1] > 160 and pos[1] < 320):
                                print("Scanning again with same settings")
                                SetFileType()
                                scan_again = True
                            elif (pos[0] > 240 and pos[0] < 480 and pos[1] > 160 and pos[1] < 320):
                                print("Don't scan again, close scanner and go back to start")
                                scanner.close()
                                scan_again = False

                            else:
                                print("AREA OUTSIDE MODE BUTTONS PRESSED")
                                scan_again = True

All of the print statements are there to leave info on the console for troubleshooting purposes. You can also exit out of the program at any screen by hitting the ‘z’ key (if you have a keyboard connected).

Here’s how it looks while it’s running (note that you have to be booted into the GUI for it to work):

And that’s pretty much what it does at this point. You can probably tell I’m not much of a UI designer, either.

I also needed a different case for this to work, because the case that came with the display only fit the Pi B+/2/3 and not the Pi 4. I came up with a pretty simple case that I’m pleased with, and the display socket fits onto the GPIO pretty much perfectly. I will probably change it again at some point but for now the display is shielded from the heat the Pi throws off, and the Pi case is vented so it doesn’t get too hot in the first place now, either. You can find the case STL files on Thingiverse.

Here it is with the Pi fastened to the lid of the scanner with part of a sticky silicone pad (the tape will be replaced with Command hooks shortly):

To get it to run automatically when the GUI loads, you need to do two things. First, make sure that you configure the Pi to boot to the GUI and log in automatically (you can set that with sudo raspi-config). Second, edit /etc/xdg/lxsession/LXDE-pi/autostart

and add the following line at the end:

@python3 /home/pi/scan-05.py
(or whatever you called your file)

Save, reboot, and the program should start automatically when the Pi loads and logs into the GUI.

I haven’t played with overlay filesystems much, but I enabled it on the Pi with the idea that the whole thing can be turned on and off with just the switch on the power cord (or by pulling the plug) and not damage the filesystem on the SD card. So far it seems to be doing the trick but I haven’t really put it through the gears yet.

So yeah… that’s pretty much it. Still a work in progress, not the prettiest thing, but it seems to work. And now anyone in the house can scan whenever they want to, and we don’t need to worry about finding the scanner and hoping it works again at the end of November.

Thanks to the following people and/or resources:

  • Running something when the GUI boots: Arnold Chan, https://raspberry-projects.com/pi/pi-operating-systems/raspbian/auto-running-programs-gui
  • Setting the SANE scan resolution: various, https://python.hotexamples.com/examples/sane/-/get_devices/python-get_devices-function-examples.html

If you’ve read this far, you’re either one of those bots that scrapes content, or a human that’s pretty bored. This was a fun project, though, and I hope there’s something in here you can use!

10YQM / NT64 COB LEDs

Some months ago a local store had a whole pile of these AAA battery powered LED lamps that you could mount pretty much anywhere:

LED lamp that looks like a light switch
Back of LED lamp that looks like a light switch

They weren’t all the same – some had contacts for three AAA batteries, some had contacts for four, and there were a couple of different brands. They all looked pretty much the same, though, and were surprisingly bright (some had cheap batteries in them from the factory I guess) and really, REALLY inexpensive.

I bought a few of them, took them home, and puttered around with them. Some had a resistor in them and some didn’t. I tried running one off my bench supply set to the right voltage and they started to smoke. Even with a set of AA batteries instead of AAA they got pretty hot. So… not only were they cheap, they were counting on folks using AAA batteries with a particular current capacity/internal resistance. Not sure if that’s clever engineering or just super cheap.

I was really impressed with those Chip on Board (COB) LEDs, though. Even with the batteries from the store in them they were crazy bright. I went back to the store and bought out the rest they had, thinking I’d take them apart and use the LEDs for other purposes.

And then they sat.

Fast-forward to a couple of days ago. I have this snazzy little gooseneck-mounted magnifier and wanted to turn it into an illuminated one. I tried a couple of different approaches and wasn’t happy with them. Then I remembered those switch lamp things.

Taking them apart is easy, it’s just six screws on the back (two are under the sticker), and then the whole thing pretty much falls apart until you’re left with this:

LED lamp disassembly showing internal parts

A slide switch, some wires, two COB LEDs mounted to a plastic reflector, and sometimes a resistor (this one didn’t have one).

Snip the wires and the plastic “rivets” and the LEDs come off easily:

LED lamp showing removed switch and LEDs

Notice how there is one red wire, one blue wire, and two white wires? There is a positive and negative terminal on both ends of the LEDs. Both positive terminals are connected, as are both negative terminals, so you can chain the LEDs together however you want. In this lamp’s case, they were wired in parallel.

There are two markings that I could find on the LEDs in this lamp and on several others that I tried – “10YQM” and “NT64”:

COB LED 10YQM
LED COB NT64

Each COB has ten LEDs in it and the back of the COB is a nice little aluminum strip that will carry away some heat. Unfortunately, I had no idea how much heat, or how much current would generate how much heat, because I couldn’t find any datasheets (or even information) on these devices.

So, bench time! I set up an LM317 as a constant current source and, starting from 50mA, slowly increased the current until the voltage across the device stabilized, then increased it some more until the aluminum got just slightly warm to the touch. What I ended up with were the following numbers (tested on six of these devices):

10YQM/NT64 COB LED
Forward voltage: 2.7-2.8V @ 150mA
Forward current (sustained, no external cooling required): ~150mA
Maximum current tested: ~350mA for less than 10 seconds (got hot pretty quick)

I’ve had them running for five hours at a time at 125mA with no problem, and at that power, four of them are plenty bright enough for a desk lamp:

Desk lamp made with 4x LED COBs
It’s too bright to look at it with my eyes but I like this picture because the short exposure shows the individual LEDs in the COB.

They’re wired in series and held on with some masking tape (you can see it in the picture) but I’m printing a proper holder for them right now. Maybe it’ll be the subject of another post.

Anyway, if you have any of those LED switch lamps or some of those 10YQM/NT64 COB LEDs, good news – they’re nice and bright and easy to use!

Removing IR Filter From ESP32-Cam, Part III

Okay, so it’s been a while but I finally got a chance to tinker with the ESP32-Cam in the dark again. 🙂

I’m using the same kind of IR illuminator that I used last time. You can get them all over the place. These little guys:

IR Illuminator for Raspberry Pi Camera
Image taken from https://www.amazon.ca/s?k=raspberry+pi+ir+illuminator

I went back over what I’d done last time and looked at my notes that I’d made after. The last time, I thought I’d play it safe and use a constant-current source to power the LED. I’ve had good results in the past using an LM317 to do this quickly and cheaply, so that’s the way I went, limiting the current to 125mA.

Turns out, though, that I was underpowering the illuminator because not only is it capable of more than 125mA, it was only getting around 2.4V because I was powering the LM317 off a 5V supply instead of the 12V I should’ve used.

This time around, I changed how I powered it. After a bit of poking around, it seems the illuminator runs on a 3.3V supply. I wanted to run it off a 5V USB pack because it’s so convenient, so I cheated and just put two 1N4001 diodes in series between +5V and the illuminator to get down to around 3.3V:

Two diodes in series on a breadboard

I also ended up adding a 1uF tantalum capacitor between the GND and +5V pins of the USB connector to cut down on some of the noise.

There is a tiny potentiometer on the module that supposedly controls at which point the module will turn on the LED, but it seems that it controls the current through the LED and has very little (if anything) to do with the amount of light on the photoresistor. Turning it all the way in one direction (clockwise, I think) gives a current of about 65mA through the LED, while all the way the other direction gives a current of around 420mA.

Unless you attach a heat sink, DO NOT RUN THESE IR ILLUMINATORS AT FULL POWER. It didn’t take too long before I could smell that all too familiar burning electronics smell, and the module itself failed shortly after.

I took another one, covered the photoresistor, and dialed it in to about 300mA. In this case, with unrestricted natural convection, it was able to run for over five hours before I turned it off. It was hot to the touch, but not unbearably so. Yours will probably be different, so keep an eye on it!

Here’s what happened with the current as the time went on:

Current reading through USB port
Current reading through USB port
Current reading through USB port
Current reading through USB port

So in the first 20 minutes or so the current climbed from 295mA to 325mA and then stayed there for the nest four hours and 50 minutes. For the entire time, the voltage across the module was 3.2V:

Voltage across IR illuminator

So, 325mA at 3.2V works out to 1.04W. That’s a lot better than the 0.3W from the previous setup. The advertising for these modules claims 3W, but I think it’s assuming you’re using both at the same time (they all come in 2-packs) and are running them with the pot dialed to full power.

Yes, there are problems with how I did this. Using two diodes isn’t a great way to “set” the voltage. Those USB meters are notorious for being inaccurate, too. So again, if you decide to try this, keep an eye on your stuff!

So the big question then, is: does running the LED at 3.5x the power make much of a difference? I think it does. Here’s a look from the dining room table with the LED covered up…

Living room, no illumination

… and uncovered:

Living room with IR illumination

Note that surface C is about six feet from the camera. A is ten feet, and B is about 22 feet. This is noticeably better than the last set of tests – the pictures I took inside the hallway with the closed doors last time confined the IR and kept it bouncing around. In this picture, the room is wide open and the pattern on the wall over 20 feet away is still easy to make out.

Here we are back in the hallway like last time. Here it is without IR:

Dark hallway, no IR

… and with IR:

Dark hallway with IR illumination

For reference, here’s what the camera saw when I did this test back in May:

ESP32-Cam

The picture from today with the illuminator running 3.2V@300mA is noticeably brighter and less grainy. This is a good thing!

And I braved a near-solid wall of mosquitoes to try the ESP32-Cam again outside on the deck steps. Here’s a picture with no IR (it’s a little blurry because I was waving my arms trying to keep from being exsanguinated:

Yard at night, no IR

Unfortunately, the bird feeder has moved around the yard a few times since May, so I wasn’t able to use it to compare with last time. However…

Yard at night with IR on

… this time I can see THE FENCE. And the garden. And the solar panel in the garden. And the little weather station thing. Even the eavestrough on the neighbour’s house! Here’s what it looked like last time:

ESP32-Cam

Quite the difference! I’m also only using one of the illuminators – they come in pairs. Twice the output won’t correspond to twice the brightness in the camera, but it will certainly lighten things up even more.

The ESP32-Cam with the IR filter removed and this particular illuminator do not put on a stellar performance, but I’m comfortable saying that they works well indoors, and they’re decent outside, too. Not good enough to, say, catch a license plate of a car driving by, but just fine within six to ten feet.

I still plan to put one of these outside, but getting a wifi signal out there is pretty tricky. I’ll either have to move the router or add something else to the network here. That might end up being a whole other project.

Thanks to everyone who commented and sent messages – I appreciate the feedback!

Removing IR Filter From ESP32-CAM, Part II

**UPDATE (2020-05-08): I did a little bit of math and realized I was only running the IR illuminator at about a quarter of a watt. After a bit more poking around and squinting at part numbers, I’m pretty sure it can run at almost 10x the power. I will look into this further sometime soon.**

Shortly after my previous post on removing the IR-cut filter from the OV2640 camera that comes with the ESP32-CAM, I put the ESP in a safe place… so safe that I lost it.

I found it again a couple of days ago and finally got to tinkering with it this evening, with the plan to see how well it worked in the dark.

I changed the following settings from the defaults for all of the pictures taken with the ESP in this post:
– Frame size: 1024×768 (XGA)
– Gain ceiling set to 3
– Special effect set to 2 (greyscale)

Without the filter, the colours were all wrong and it was a little harder to focus. This is normal but a little annoying, so I set the image to greyscale. Here’s the first image I took, looking at the bird feeder as the sun was on its way down on the other side of the house:

ESP32-Cam with no IR filter
I tried to set the focus at where the bird feeder is. Not sure if I was entirely successful…

It’s not too bad. Things look a little weird – the trees, sky, and grass all look like they’re from a really cheap dream sequence. Two hours and fifteen minutes later, the sun had set but there was still plenty enough light for the camera:

ESP32-Cam

Twenty minutes later, the streetlights were all on, there was very little glow in the sky, and the image started to get pretty noisy:

ESP32-Cam

Another 25 minutes and it was dark:

ESP32-Cam
The bright dot is a light the next block over

So, with ambient light in the suburbs at night, the ESP32-Cam isn’t a stellar performer. This is not a high-end CCD rig.

I have a couple of inexpensive little IR illuminators but they’re so inexpensive that they didn’t come with specifications and I can’t find datasheets for them. I ended up using an LM317 to limit the current to about 125mA. I’m pretty sure they can go higher but figuring out how high is a project for another day.

IR Illuminator
Yep, one of the many, MANY illuminators out there that look just like this.

Here’s what that thing running at 125mA did:

ESP32-Cam
Well, I can see… something. Not much, though.
ESP32-Cam
After shining it around to find the best angle, I can juuuust make out where the bird feeder hook comes out of the ground.
ESP32-Cam
Yep, it’s definitely on…

Not so good. Or… was it not too bad? It’s not like I went out there with one of those 400 LED yard illuminators – this was a single diffused LED that’s pulling just over half a watt. Maybe I should revisit this with something that puts out a little more oomph.

The next thing on my list to check was how well it worked indoors. To test this, I put the camera and illuminator at one end of a hallway, closed the door, and turned off all the lights. My phone picked up a bit of light under the door at the far end of the hall, about 12 feet away:

Dark hallway with phone

And here’s what the ESP saw:

ESP32-Cam

And with the same illuminator (at the same power) turned on:

ESP32-Cam
You can even see the yellow sticky note I put on the door. Can’t see the smiley on it, though.
ESP32-Cam
My shirt is black under normal light.

Having the IR confined to an area where it can bounce around and not just be lost to the night makes a big difference. With a better (or more than one) source of IR, the ESP32-CAM may well make a decent indoor night camera. Tweaking some of the camera settings may improve things, too.

So… not the best performer, but for the cost (I think I paid nine dollars for this particular one, camera and all) and the sheer number of features that the ESP32 series has, I find it a pretty attractive little device.

I think the next ESP-ish things I want to look at are getting some pictures of the animals that wander through the yard at night, and maybe trying this particular experiment again with a bigger/better IR illuminator.

Recovering A Lost mdadm RAIDset In A Raspberry Pi-Based RAID Mirror

My previous post went over how to replace a failed USB-attached RAIDset member in a Raspberry Pi system running mdadm. Swapping out a failed RAIDset member is pretty simple, but what if the Pi or the micro SD card in the Pi fails?

Well, good news – mdadm stores metadata on the RAIDset members so even if you lose the RAID configuration on the SD card, you can recover it by scanning the disks.

I’m going to use the same Pi 4 (2GB) and two 64GB SanDisk USB sticks that I used in the previous post. They’re set up as a two disk mirror.

To simulate an SD card failure, I killed power to the Pi without shutting it down gracefully, then removed and re-imaged the card before putting it back into the Pi and powering it up. Here’s how to get your data back.

First things first – flash the appropriate Raspbian image to a micro SD card and set up your Pi like you would normally. Don’t forget to go through raspi-config and do the sudo apt update and sudo apt upgrade thing!

Once your Pi is up and running, fully up to date, and configured the way you like it, then shut it down, reattach the USB drives, and start it up again. Once it’s booted, check dmesg to see if the Pi can see the USB drives. The first USB drive will be called sda, the second will be sdb, and so on:

In my case, I only have two disks, /dev/sda and /dev/sdb and they’re both showing up. If you’re not seeing the disks, check connections and power and fix the problem before going on.

Using cat /proc/mdstat is the easiest way to check the status of your RAID array, but if you run it now, you’ll get nothing:

That’s because mdadm hasn’t been installed yet. Install it with the following:
sudo apt install mdadm

Once it’s installed, scan the disks on or attached to the Pi and create a RAIDset with any member disks found, using
sudo mdadm --assemble --scan --verbose

Notice that it found both /dev/sda and /dev/sdb, and it added them to an array.

Check the status of the array with cat /proc/mdstat:

It’s showing as [UU], which means both members are online and working properly. The array didn’t even need to resync!

Now, check the /etc/mdadm/mdadm.conf file to see if the configuration has been automatically updated:

Under “definitions of existing MD arrays” you can see the array is showing up. Also notice that it’s still called “DEV-04:0” – DEV-04 was the name of the Pi before I wiped out and rebuilt the SD card, so chances are it has the right data.

Reboot your Pi to see if it (and the array) come up cleanly. This isn’t a necessary step but I like doing it to make sure things are working and I haven’t forgotten anything.

Run blkid to see if /dev/md0 is listed:

So far, so good. Now, edit your /etc/fstab file, adding an entry for /dev/md0 but use the UUID instead of the device name:

I’m using /mnt as the mount point but it can be anywhere you want.

Once you’ve saved the file (and if everything is working properly), you should be able to mount the drive and use it with sudo mount -a

In this case, boring is good

If there are no errors and the system doesn’t hang or do something else weird, things may be working! Run df to see if /dev/md0 is mounted:

Hopefully you’ll see something similar. Check out the mounted directory and see if your data is there:

It’s there… it’s all there!

Depending on how you had things set up, you may need to change or set some permissions, but if it mounts and you can browse to it, that’s a good sign. Reboot the Pi once more to make sure the array and mounting work as expected, and if so, then everything should be good to go!

You can also use this method to move a RAIDset from one machine to another, although you should really, REALLY back up your data before you do that.

Actually, getting a backup of your stuff periodically isn’t a bad idea either. mdadm is mature and pretty stable, but a RAIDset with redundancy is no substitute for backups!

Replacing A Failed USB Disk In A Raspberry Pi-Based RAID Mirror

My previous post went into how to create a simple but functional NAS with a Raspberry Pi 4B and two USB-attached SATA disks. In the two weeks or so that it’s been running, the NAS I built has performed very well and has been reliable (hopefully I won’t regret typing that).

But what to do WHEN a disk fails? Disks fail – even that fancy new enterprise-grade SSD that cost an arm and a leg will fail at some point. The good news is that if you’re using mdadm to provide some kind of redundancy with your disks, things should still be working if a disk fails. The bad news is that unless you’ve got a RAIDset that can specifically tolerate more than one failure (like RAID 6), you need to replace that failed disk ASAP.

I’m confident that I’ll be able to recover from losing a disk in my shiny new NAS, but I’m not one to tempt fate so I built another RAIDset with a spare Pi and two 64GB SanDisk USB sticks to play around with instead. They’re slower than the disks so things like the speed the RAIDset syncs back up is going to be different than in my previous post.

So here’s the setup – it’s a Raspberry Pi 4B (2GB) with two 64GB USB flash drives in a RAID 0 (mirror) configuration.

Here it is, working properly, with the output of cat /proc/mdstat:

and checking to see if it’s mounted using df:

To simulate a disk failure, I removed one of the USB sticks while everything was running. Here’s the output of dmesg showing the disconnection and that mdadm is soldiering on with only one disk:

Looking at the list of USB-connected devices only shows one SanDisk device:

And now the output of cat /proc/mdstat is showing a failed disk (note the “U_”):

The good news is that yes, /dev/md0 is still mounted and usable, even though it’s in a degraded state.

I reformatted the USB stick on my Windows PC so the data that was on there was lost, then reconnected it to the Pi:

There are two SanDisk devices again.

And here’s the output of dmesg again – you can see the time difference between the failure and when the “new” disk was connected:

Note that the messages both of the failure and of the newly connected USB stick show them as sdb. It could just as easily have been sda, so make sure you check to see which one failed – and, more importantly, which one didn’t!

So now there are two disks connected again, but only one of them has the RAIDset data on it. In this case, sda is the one with the data that needs to be mirrored over. Again, it could’ve been sdb. For one last check, get the output of cat /proc/mdstat again:

Notice it says sda – that means that sda has the data we want to mirror over to the other disk, which, as the previous output of dmesg showed, is sdb.

If you are replacing a failed RAID member, the replacement must be the same size or larger than the failed member. That goes for any kind of RAID level and any type (i.e. disk mirroring or partition mirroring). Keep in mind that not all disks of the same stated capacity will actually have the same capacity, so make sure you do a bit of research before going out and spending your money on a new disk that won’t fit your current array!

Now that the disk is reconnected and showing up, copy the partition layout from the existing RAIDset disk to the new disk with the following command:

sudo sfdisk -d /dev/sdX | sudo sfdisk /dev/sdY

In this case, the existing disk is /dev/sda and the new disk is /dev/sdb:

This step isn’t needed if you’re mirroring disks (as opposed to mirroring partitions), but it’s a good idea to do it anyway – if there’s an error here, you certainly don’t want to go any further until you’ve fixed the problem.

If sfdisk worked and didn’t give you any errors, then you’re ready to add the new disk to the RAIDset with the following command:

sudo mdadm --manage /dev/md0 --add /dev/sdY

Where sdY is the new disk – in my case, sdb:

If you didn’t get any errors, run cat /proc/mdstat again and you’ll see your RAIDset is rebuilding:

Notice how it now shows that there are two active elements in md0sdb[2] and sda[0]? That’s a good sign. Keep checking every once in a while to make sure the recovery is progressing.

Once it’s done, the RAIDset should be showing as all “U” again:

If you see that, everything’s rebuilt and your RAIDset is ready to handle another disk failure.

Hopefully you never need to use this information, but if you do, I hope it helps!

Raspberry Pi 4-Based NAS Using USB-Connected Disks

Our little DNS-323 has been rock solid for the last decade but it’s getting long in the tooth and it’s just pokey enough to be annoying when I’m trying to do things on it. After a bit of consideration, I decided to see if I could build one with a Raspberry Pi.

It didn’t take a lot of research to find out that the Pi 1 through Pi 3+ aren’t particularly suited for NAS work. They may have the CPU horsepower, but with the on-board Ethernet and USB sharing the same USB2 port, their performance is reportedly not all that great.

The Pi 4, on the other hand, has entirely different hardware; the USB ports (of which there are two USB3 and two USB2) have their own controller, while the on-board Ethernet has its own controller. This makes for an ENORMOUS improvement in the performance of both USB:

Image from (2020-03-23): https://magpi.raspberrypi.org/articles/raspberry-pi-4-specs-benchmarks

and Ethernet:

Image from (2020-03-23): https://magpi.raspberrypi.org/articles/raspberry-pi-4-specs-benchmarks

With numbers like that, I thought it would be worthwhile to try building a little RAID1/mirrored home NAS around a Pi 4. Here’s what I used:

  • Raspberry Pi 4, 2GB model, qty 1
  • Official Raspberry Pi 4 power supply, qty 1
  • 32GB Sandisk UHS-1 MicroSD card, qty 1
  • 4TB Western Digital Blue 3.5″ hard drive, qty 2
  • Vantec NexStar TX 3.5″ external USB3 enclosure, qty 2
  • 5ft Category 5e patch cord, qty 1

I went with mirroring two disks (RAID1), so that is what I’m going to go through here. If you want to set up a single disk, or set up something like a four-disk mirror, or RAID5/6/1+0, you can use the same software I did but you’ll have to do a bit of research into the settings.

Oh, and if you are going to create a RAIDset with more than one disk, make sure they’re all the same size, otherwise the mirror will only be as large as the smallest of the disks that are part of the RAIDset!

I had originally planned to use openmediavault to mirror the disks and create the network shares, but unfortunately it doesn’t support USB-connected disks. That’s what I get for not reading enough before I buy stuff, I suppose. With omv out of the picture, I decided to try mirroring the disks and set up the network shares myself. Here’s how I did it:

Part 1: Set Up Your Raspberry Pi

I’m not going to get into this because there are already a ton of sites out there that will show you how to do this (and describe it better than I can). I built my NAS with the Raspbian Buster Lite image, dated 2020-02-13. Do not use wireless (don’t bother with a wpa-supplicant.conf file), but make sure you enable ssh, go through the raspi-config menu and don’t forget to apt update and upgrade!

Oh, and CHANGE YOUR PASSWORD!

Part 2: Assemble And Format The Disks

This part’s pretty easy. Install the hard drives in the enclosures, connect them to your PC, and using the software of your choice, remove any existing partitions on the disks and create a single NTFS partition (or ext4 if your PC is a Linux machine) that uses the entire capacity of the disk. Once that’s done and you get no errors, safely remove the enclosures from your PC.

Part 3: Put Together And Connect Your Hardware

This part’s pretty easy too. Connect the USB disks to the Pi and turn them on. If things are working properly, the physical enclosures and the disks will be present. To check if the Pi sees the enclosures, type lsusb:

Two USB to SATA devices, check.

To see if the actual spinning hard drives have been detected, type dmesg | grep sda for the first enumerated disk, and dmesg | grep sdb for the second:

The second disk (/dev/sdb) should look pretty much the same.

Do not go any farther if the output of either of those commands doesn’t look correct, or if the disk capacity listed is different than you expect. Go back, check all of the parts and connections, and try again.

Part 4: Set A Static IP On eth0

You should at this point already have a cable connecting your Pi to your router or a switch. If you’re using wireless… well, I suppose you can do that if you really want or need to, but you’re going to be making your Pi do its work with one foot in a bucket. Use that RJ-45 jack and get yourself some nice clean Gigabit Ethernet goodness.

To set a static IP, use your favourite editor to edit the /etc/dhcpcd.conf file. Make sure to use sudo so you’re editing it as the superuser. Go right to the bottom of the file, and add the following lines:

# NAS Static IP for eth0
interface eth0
static ip_address=X.X.X.X/YY
static routers=Z.Z.Z.Z
static domain_name_servers=A.A.A.A B.B.B.B

Where:

  • X.X.X.X is the static IP address on your network that you want your NAS to be reachable at
  • YY is the CIDR representation of your subnet mask (most home or small businesses will be /24)
  • Z.Z.Z.Z is the IP address for your gateway/router
  • A.A.A.A is the IP address for your primary DNS server
  • B.B.B.B is the IP address for your secondary DNS server (if you have one)

Make sure that you type everything exactly. Even if you only have one router and one DNS server, you still need to type static routers and static domain_name_servers with the “s”.

Once you’ve finished setting the dhcpcd.conf file, reboot your Pi. Once it comes back up, see if you can ping other devices both inside and outside your network. Then try to do another apt update and upgrade to see if your Pi can talk to the Raspbian repositories. If not, go back over the file and make sure the changes you made were saved. Also check to make sure everything is spelled correctly (remember it’s case sensitive).

Part 5: Mirror The Disks

Congratulations! If you’re here, that means you have successfully set up your Raspberry Pi, can see it on the network, and have two hard disks connected via USB. You are now ready to do something not a lot of other people do – use a Raspberry Pi to make a RAIDset out of a pair of USB-connected disks. Fortunately, the software (like the hardware) has made leaps and bounds since the last time I tried it and it’s pretty easy to set up.

First, let’s make sure the right stuff is installed:

sudo apt update
sudo apt upgrade
sudo apt install mdadm

Since you already made sure the disks were working in Step 2, you can go ahead and create a RAID1 mirror. In my case, I didn’t care about the partition size so I used the entirety of both disks with the following command:

sudo mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sda /dev/sdb

In this case, mdadm is being told:

-- create : Make a new RAIDset
-- verbose : Show what’s going on while the command is running
/dev/md0 : The name of the RAID device you’re creating
--level=mirror : Create a mirror (RAID1)
-- raid-devices=2 : How many disks will be used
/dev/sda /dev/sdb : The names of the disks that will be used

Again, this is how I set up my own little two-disk mirror. If you have a different number of disks or want to set up a different kind of RAIDset, the syntax is pretty much the same but the options are different. You may also want to use particular partitions instead of entire disks like I did. Check out the mdadm man page.

You should now see the lights on the enclosures blinking furiously and/or be able to feel/hear the the hard drives doing something. Depending on the kind and size of disks you have and the type of enclosure, creating the mirror and syncing it up may take up to a day. You can check the progress with the following command:

cat /proc/mdstat

which should give you output something like this, which shows you the status of the mirror sync:

Personalities : [raid1]
md0 : active raid1 sda[3] sdb[2]
3096885440 blocks super 1.2 512K chunks 1 near-copies [2/2] [UU]
[>....................] resync = 0.2% (61504/3096885440) finish=398min speed=14298K/sec
unused devices: <none>

Once it’s done, instead of the above, you should see something similar to the following when you run the same command:

pi@PI-0:~ $ cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sda[1] sdb[2]
3906885440 blocks super 1.2 [2/2] [UU]
bitmap: 0/30 pages [0KB], 65536KB chunk

Note the line with the [UU] at the end. Each U represents an active and healthy RAID disk. If you run cat /proc/mdstat and you see a _ (underscore) instead of a U, there’s a problem with a disk that requires your immediate attention.

Now that the RAIDset is built, you need to save its configuration so your Pi knows what to do with it when it boots:

sudo -i
mdadm --detail --scan >> /etc/mdadm/mdadm.conf

exit

Now confirm it was saved:

cat /etc/mdadm/mdadm.conf

and look for the line that says something like this (obviously your UUID will be different):

ARRAY /dev/md/0 metadata=1.2 UUID=061a78a9:ceadf64b:b124c1d4:7e35ae85 name=PI-0:0

Now reboot your Pi, and once it comes back up, use cat/proc/mdstat and blkid to see if everything’s okay:

Seeing [UU] and /dev/md0 is a good sign

Part 6: Create A Filesystem

Now that the disks are mirrored, it’s time to put a filesystem on them. I use ext4, and I’m creating a filesystem on the RAIDset, NOT the physical disks, so the command is:

sudo mke2fs -t ext4 /dev/md0

This may also take a little while. Once it’s done, edit the /etc/fstab file so that the filesystem on the RAIDset will automount at boot. I use the /mnt directory as the mount point, here’s what my fstab file looks like:

Notice that the UUID of the partition is the same as the UUID for /dev/md0 in the output of blkid.

Type the following to see if your fstab file is set up right:

sudo mount -a
df

You should see something like this:

In this case, /dev/md0 is mounted at /mnt and everything looks good. The use is at 12% because I’ve already been using the NAS – yours will probably say 0% or 1%.

Now, shut your Pi down, then turn off the disks. Turn on the disks, wait for them to spin up, and boot up the Pi. Run those two commands again and make sure everything looks good before going any further.

Part 7: Set Up A File Share With Samba

Samba is a mature, stable, and very useful batch of software that makes it pretty easy to create simple network shares. It may already be installed, but just to be on the safe side:

sudo apt install samba
sudo apt install samba-common-bin

Once it’s installed, you’re going to need to configure it by editing the /etc/samba/smb.conf file. I’m only going to set up one network share for now, and if you’re new to Samba, I suggest sticking to one share until you’re familiar with it.

Before we edit the file, though, we need to create the directory that Samba will use to share over the network:

sudo mkdir /mnt/NAS_FILE
sudo chown pi /mnt/NAS_FILE
sudo chgrp users /mnt/NAS_FILE

Do NOT use /mnt as the directory for your file share – always use a directory that resides on the device you’re mounting. If for some reason /dev/md0 doesn’t mount properly, you may end up writing data to and filling up the SD card instead of using the disks!

The default smb.conf file contains a number of examples, including one for a network share and one for a print share. Copy the default file:

sudo cp /etc/samba/smb.conf /etc/samba/smb.conf_OLD

Now, with sudo, use your favourite text editor to open /etc/samba/smb.conf and go to the bottom section of the file, labelled “Share Definitions”. Delete everything in that section and replace it with the following:

[NAS]
comment = NAS Fileshare
path = /mnt/NAS_FILE
browseable = yes
read only = no
writable = yes
create mask = 0775
directory mask = 0775
valid users = pi

So the “Share definitions” section in my smb.conf file looks like this:

Notice how the “valid users” section has the name “pi” in it – you can change that to anyone you’d like (or have more than one user on that line), but for each user on that line, you’ll need to create an account on the Pi for them. I just stuck to the pi user because I wanted to keep things simple to start.

To test your smb.conf file, run the following:

testparm

The “Loaded services file OK” is a good sign that your smb.conf file has no obvious errors in it. If you don’t get that message, go back through the file and make sure everything is spelled properly, etc.

Now, for each user account you want to grant access, you need to run the smbpasswd utility to set them up in Samba. It will ask for a password, and that password really, really should really be different than the password that’s used by the user to log into the Pi itself!

To add someone to the Samba system:

sudo smbpasswd -a USERNAME

To disable someone’s Samba account:

sudo smbpasswd -d USERNAME

To re-enable someone’s Samba account:

sudo smbpasswd -e USERNAME

In my case, I ran:

sudo smbpasswd -a pi

and gave it a password that was different from the password that I use for logging in with the pi user.

At this point, Samba is installed, you’ve created a directory on the RAIDset that Samba will use for the file share, you’ve edited the smb.conf file, ran smbpasswd for every user that’s listed in the smb.conf file, and tested your configuration with testparm. It’s time to restart Samba so it loads the new configuration:

sudo service smbd restart

If you don’t get any messages or errors, things may actually be working!

Part 8: Access Your New NAS

Finally, the payoff – your own home-built NAS! How you will access it depends on the operating system on the computer you want to access it with. If you open your file or network browser, it may automatically show up. Otherwise, you will have to browse to it. Open your file or network browser and browse to the static IP address you set way back in Part 4.

In Windows, it should look something like this:

In Ubuntu, you may have to enter smb:// before the address:

When you try to open the NAS share, you should be prompted for a username and password. The username will be:

localhost\USERNAME, which in my case was
localhost\pi

…and the password will be whatever you set it to with smbpasswd.

Some of my friends and family may disagree with this statement, but I like it when things are organized. To help with this, I designed and printed a lightweight frame that holds two disks and a Pi, and has several holes for 5mm M3 screws to fasten things like cable management or velcro or whatever to it. Here’s how mine turned out:

NAS Frame

I put the model on Thingiverse, it’s at https://www.thingiverse.com/thing:4235287

As for performance… this setup is much more responsive and can transfer files to and from the disks much faster than the old NAS. Disk fragmentation can slow things down, and the old NAS is a decade old, but it was only about half full so that shouldn’t have been too big a problem.

Transfer rates to and from the Pi are faster than my home wireless is, and browsing the directories on the file share is no different than browsing the directories on my PC. Nice and quick.

150Mbps isn’t too shabby at all! The best I could get out of the old NAS was around 70Mbps. Bottom line – a Pi 4 with two external USB3-connected hard drives makes a serviceable and reasonably fast NAS for home or small business use, although there are security considerations that need to be addressed prior to using it out in the real world.

So that’s the deal. I did a bunch of torture testing when I first set things up, and things recovered gracefully. I will do another post soon to discuss how to fix a mirror if there’s a disk failure or if you need to recover the array entirely due to a Pi failure.

I hope you found this useful!