ECE434 Project-Image recoginition with openCV and Siri

Team members: Haoxuan Sun, Heda Wang

Executive Summary
In this project, we will use OpenCV to do image recognition. A servo will make the camera centered at the desired target. Also, it is controlled by Siri through shortcuts and ssh.

Currently, this project only supports three colors. But you can easily add colors you want to find. This project is limited by the processing speed of the beagle bone. Because the processing takes a long time for each frame, we can only point to the color instead of tracking. With enough optimization and computational power, this can be easily turned into an image tracker.

To achieve this project you need:

1. BeagleBone Black 2. A servo 3. A USB webcam 4. LED or other colorful things for target 5. (optional). iPhone or iPad Supports Siri

A demo video is available on YouTube: https://youtu.be/_aZgPR_QLXI

Hardware
Connect the USB camera to the USB port. Connect the servo to 5V on the Beaglebone. If the servo is not powered, you might need to plug in the 5v power supply on the beagle bone. The servo signal is connected to pin 8_13

Tie or glue the servo and the camera together. We will use the camera for other use laster, so we tie the camera and servo using some wire.

Installation Instructions
We need to install openCV To install it, connect to the BeagleBone and type:

sudo apt install python-opencv

sudo pip3 install opencv-python

and if error occurs, type:

pip3 install opencv-python sudo apt-get install libcblas-dev sudo apt-get install libhdf5-dev sudo apt-get install libhdf5-serial-dev sudo apt-get install libatlas-base-dev sudo apt-get install libjasper-dev sudo apt-get install libqtgui4 sudo apt-get install libqt4-test Connet your beagle bone to your router though a cable Also, make sure there is shortcut on your ios device. You can install it from APP store. Click create add shortcut and scripting. Then select the Run Script Over SSH. Touch the show more to see all the option. Go to your router config page and find your beagle bone's IP address. Enter the IP address to "Host" and the port as 22. The user is usually Debian and also enters the password. In the script enter sudo python3 red.py. Click the + under the block you just added and select scripting again. Select show Notification and configure it as "I am pointing red". Finally, rename the shortcut as "point red" or whatever you want Siri to recognize. If you want to deal with more color, just copy this shortcut and change the code it run.

After configer all the shortcut the screen should look like this. figure here

User Instructions
Activate Siri by pressing the home button or saying hey Siri. Tell the Siri "point red/blue/green" and wait for the servo and camera to point at the color. After the process is done, there will be a notification pop up in your device indicating it is done.

Highlights
This is a demo video with our explaination. https://youtu.be/ch4CfHEJSlM

Theory of Operation
1. Scanning for color. Because the image processing is so slow and our servo is not very accurate. We choose to scan several angles and find the best angle that aligns the camera and the color. You can change the scanning step in the code. But more steps means there is more time until you can see the result.

2. Image processing and recognition

opencv ———————————— Haoxuan Sun write this part

recognition

After we get a black and white picture with only the object we care about, we put every pixel above the threshold in the image into an array. Then we find the middle pixel. This is likely to be around the center of the object.

Work Breakdown
Heda Wang: Learn and config Siri shortcuts, Wrote the servo code and the scan algorithm, Integrate the servo code with image recognition code.

Haoxuan Sun: Install Open-CV, Wrote the image processing and recognition code. Figure out how to find ip address on campus network.

Other things we done them together.

Future Work
Our initial plan was to make the servo track the object. But the image processing speed is so limited. Every frame need several seconds to process. If we can optimize the algorithm, we can achieve track the boject.

Also, The short cut on Siri can accept parameter input. With more development, we could let siri accept any color input with only one python file.

Conclusions
There is a lot of potential with the Siri shortcut. Adding more fun things to the shortcut will make this project connect better to ios devices.