.jpg)
.png)
%20(1).png)
%20(1000%20x%20500%20px)%20(1).png)
November 23, 2017
Object Detection on a Raspberry Pi
Image recognition has become a part of our daily lives, and the technology behind it is advancing at a steady pace. We thought it'd be cool to use the increasing speed and tiny size of lightweight computers like the Raspberry Pi, as well as the efficiency and portability of machine learning libraries such as Tensorflow, to create a standalone, handheld object detector.
The first step is to find out whether running live object detection on a small device such as the Raspberry Pi is viable; until recently the technology to detect multiple objects at the speed we require just wasn’t there. Luckily for us, the folks at Google Brain were kind enough to open-source their object detection API, which does just this.
The use cases for a portable object detector are many and varied - there are places where having a full PC set-up isn't viable, and where an internet connection may not be available for outsourcing the detection to the cloud. The Raspberry Pi is so lightweight that you can even mount it on a drone.
.png)
Initial Setup
To get started with object detection on the Raspberry Pi, you of course need to have a Raspberry Pi. We used a model 3, running Rasbian Jessie. You also need a camera attached to the Pi. Once the pi is up and running and connected to a monitor (or through SSH), you can open up the terminal and install the pi camera by entering the following commands:
You can then test out the camera by running the raspistill command tool like so: raspistill –o filename.jpg
Once we’ve confirmed that the hardware is working, we have to make sure we’ve got the Python Package Index installed:
This will allow us to install most of the required packages, though because the OS we’re using has a slightly limited package index, we’ll have to do a couple by hand.
Software Installation
There are a number of libraries you need to install to get object detection up and running, the main ones being Tensorflow, OpenCV, and the Object Detection API. Installing these on the Raspberry Pi is a little different to installing them on desktop Unix-like environments, so take care that any tutorials you’re following are going to be compatible with the version of Rasbian that you’re using.
Tensorflow
Tensorflow is an open-source machine learning library developed by the Google Brain team. Tensorflow is the core of our object detection, and should be installed first. Regular Tensorflow doesn’t run on the Raspberry Pi, so we’re going to use Sam Jabrahams TensorFlow on Raspberry Pi 3.
Detailed instructions are available on the Github page, but the main commands required are as follows:
If you run into any errors, check out the official Github page.
.png)
Object Detection API
Object detection comes as part of the official Tensorflow research models. Its purpose is to detect multiple objects in single images.
Clone this repository somewhere handy:
git clone https://github.com/tensorflow/models.git
We need a variety of libraries to run object detection, so we first need to install all of these:
If any of these fail, you may need to download the wheel file manually. These can be found on pypi.python.org.
I needed to do this for lxml, as well as a couple of other dependencies for OpenCV. These packages may have dependencies of their own, so if you have any trouble installing these through Pip, go to their official installation instructions and install them manually.
Official installation instructions for the above packages:
Pillow, Lxml, Jupyter, Matplotlib
We now need to compile the Object Detection API using Protobuf. Navigate to your tensorflow/models/research/ directory, and run the following:
To run object detection, you’ll need to append two directories to your PYTHONPATH. This can be added to the end of your ~/.bashrc file, or ran manually with each new terminal you run (From the tensorflow/models/research/ directory):
Finally, test the installation with the following command:
Note: I needed to add a copy of object_detection/data into my projects src/ directory for my project to be able to find the required files.
OpenCV
OpenCV is a powerful computer vision framework, containing a huge number of algorithms for processing and analysing images. We’re going to be using it for some its simpler features, but having the full set of tools available means we can later process our images if we want to. OpenCV is perhaps one of the more error-prone libraries to install on the Raspberry Pi, so take care during this step. You should be able to find a solution on Stack Overflow for most issues, but I’ll make note at the points that caused us grief.
Initial Commands:
Install:
Warning: sudo make install can take a while.
Tip: Depending on your environment, you may need to add sudo to the start of more of the commands above. For example, if pip3 install numpy doesn’t work; try sudo pip3 install numpy instead.
If you receive an error about the GTX version you are running when you attempt to use OpenCV:
- Navigate to your matplotlibrc file in usr/local/lib/python3.4/dist-packages/matplotlib/mpl-data/
- Find the backend line, remove the #, and change the line to: backend : TkAgg
The fun stuff
If all went well, we’ve now got the Object Detection API all ready to go, and we’ve got OpenCV available to display our detection. The Tensorflow team have provided a great tutorial for getting this up and running, which I have adapted here.
We can now write a small python program to:
- Initialize object detection with a pre-trained model (a frozen inference graph)
- Stream camera frames
- Run object detection on each frame
- Output resulting image to an OpenCV window
First we import all that stuff we just installed.
We now download a frozen inference graph – this is from the COCO (Common Objects in Context) dataset. There are multiple types you can use here, of varying levels of speed and accuracy. Because we are wanting to stream object detection, and are doing so on a raspberry pi, we use the fastest one; ssd_mobilenet_v1_coco_11_06_2017. You can find the rest here.
We can now use this to create our detection graph, the final piece of Tensorflow setup before we can begin our detection.
Set up the PiCamera, and create an array to store our streaming data in.
Our main loop. This initializes each image, runs the Tensorflow session on it, and visualizes the detection boxes onto the image. It then displays the image in an OpenCV window.
Note: The bottom part of this loop where we exit on a ‘q’ press is required for the OpenCV window to work, so make sure it’s there.
Result
.png)
As you can see, the result is pretty good! The object detection isn’t quite as accurate close up – it seems to think my hand is a bed - but everything else is pretty accurate. Keep in mind that the COCO dataset we're using is trained on a set of about 180 common objects, so you may need to further train this model using your own images if you want something a bit more specific.
The above runs at about one frame per second on our Raspberry Pi, which isn’t too bad for real time object detection on such a small device. This could likely be optimised too – for example, the images could be simplified and processed before passing them into Tensorflow.
All in all, we thought this was a pretty good result for a first attempt. With a bit of optimisation, this could work in a variety of scenarios. You could mount a raspberry pi mounted inside of a robot, allowing it to navigate and recognise objects or people. You could attach a camera to the bottom of a drone, monitoring crops as it flies over. You could mount it onto a beehive, warning you of potential wasp attacks. I’m sure there are plenty of things you could do with portable object detection that we haven’t thought of, and I’m excited to see what this technology can offer in the future.