Jetson/Tutorials/Vision-controlled GPIO

< Jetson
Revision as of 10:21, 9 January 2015 by (talk | contribs) (Testing your camera)
Jump to: navigation, search

Turn an LED on/off based on whether a face is detected by a camera

This tutorial will explain how to detect a face using OpenCV, and how to turn an LED on/off based on whether the face is detected.

Testing your camera

  • Connect a USB webcam to the full-sized USB 3.0 port on Jetson TK1 (or into the micro-AB USB 2.0 port through a micro-B to female USB-A adapter).
  • If you haven't tried using your webcam on Jetson TK1, check if it works first such as by running this to show the camera preview in a graphical window:
sudo apt-get install luvcview

(If you can't see the camera stream, then try plugging your webcam into the micro-AB USB 2.0 port through a micro-B to female USB-A adapter, or looking at supported cameras on the Cameras reference page).

  • Not all working webcams seem to be supported by openCV, example programs stall with: Corrupt JPEG data: 4 extraneous bytes before marker 0xd9
  • Sony PS3 eyecam wfm

Detecting a face using OpenCV

  • If you haven't done so already, make sure you have installed the CUDA toolkit and the OpenCV development packages on your device, by following the CUDA tutorial and the OpenCV tutorial.
  • Download the OpenCV samples (they come with the OpenCV source code). You can download this by visiting "" in a browser and clicking to download "OpenCV for Linux/Mac", or if you want to download this directly from the command-line then run this on the device:

(If you don't have the wget or unzip commands, then run "sudo apt-get install wget unzip" to download & install them).

  • Make your own "faceActivatedGPIO" folder for this project, eg:
mkdir ~/faceActivatedGPIO
cd ~/faceActivatedGPIO
  • Copy the "cascadeclassifier" face detection sample from OpenCV's GPU-accelerated samples folder and build it:
cp ~/opencv/samples/gpu/cascadeclassifier.cpp .
g++ cascadeclassifier.cpp -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_calib3d -lopencv_contrib -lopencv_features2d -lopencv_flann -lopencv_gpu -lopencv_legacy -lopencv_ml -lopencv_objdetect -lopencv_photo -lopencv_stitching -lopencv_superres -lopencv_video -lopencv_videostab -o cascadeclassifier
  • Try running it with your attached webcam:
./cascadeclassifier --cascade ~/opencv-2.4.9/data/haarcascades/haarcascade_frontalface_alt.xml --camera 0

(Hit 'm' to search for just a single face instead of multiple faces, then hit Spacebar if you want to compare CPU vs GPU performance).

In case of difficulty try Testing openCV.

Testing GPIO on Jetson TK1

  • If you haven't done so already, follow the GPIO tutorial so that you can manually turn an LED on/off using root-permissions.
  • We want to control GPIO from our own code. Download Derek Molloy's SimpleGPIO library and get the 3 source files in the "gpio" folder. eg:
sudo apt-get install git
git clone git://
mkdir gpio
cp ../boneDeviceTree/gpio/*.cpp gpio/.
cp ../boneDeviceTree/gpio/*.h gpio/.
  • Let's build & run the sample TestApplication to make sure GPIO works. Remember that it must be run with root-permissions in order to access GPIO:
g++ gpio/TestApplication.cpp gpio/SimpleGPIO.cpp -I gpio -o testGPIO
sudo ./testGPIO
  • You should now be seeing your LED flash several times (or see the voltage going up & down on your multimeter).

Controlling the LED based on face detection results

  • Rename "cascadeclassifier.cpp" to "faceActivatedGPIO.cpp", eg:
mv cascadeclassifier.cpp faceActivatedGPIO.cpp
  • Now it is just a matter of combining the cascadeclassifier sample code with the SimpleGPIO library. Near the top of the "faceActivatedGPIO.cpp" file, add:
#include "SimpleGPIO.h"
  • Then near the bottom of the file after it says imshow("result", frameDisp); (line 277), add this code:
// Turn the LED on or off depending on whether a face was detected or not.
const int GPIO_LED = 57;  // GPIO_PH1 (Pin 50 on J3A1 of Jetson TK1 board)
gpio_export(GPIO_LED);    // Start using the LED pin
gpio_set_dir(GPIO_LED, OUTPUT_PIN);   // The LED is an output
if (detections_num > 0) {
    gpio_set_value(GPIO_LED, HIGH);   // Turn the LED on
else {
    gpio_set_value(GPIO_LED, LOW);    // Turn the LED off
gpio_unexport(GPIO_LED);  // Stop using the LED pin for now
  • Build the project. eg:
g++ faceActivatedGPIO.cpp gpio/SimpleGPIO.cpp -I gpio -o faceActivatedGPIO
  • And finally, run the project with root-permissions. eg:
sudo ./faceActivatedGPIO

You should now see your camera preview on your screen, and if your face is detected then your LED should be on, and if not then it should be off!

More ideas

You can improve this project in many ways, such as turning on an electro-mechanical relay that opens a door when a face is detected, instead of just flashing an LED. Or turn multiple LEDs on based on how many faces were detected or their size, by using multiple GPIO pins. And obviously, now that you know how to perform some vision-based GPIO, you could apply the same principles to activate GPIO when the full-body of a person is detected (using OpenCV's HOG sample as explained in the Full Body Detection tutorial) or when a car is detected (using the same HOG or CascadeClassifier code but with an XML file trained for car classification), or thousands of other potential computer vision or software based detections.

This tutorial is also the basis for the vision-controlled robot tutorials such as the SCOL walking robot tutorial.