ECE497 Project: Kinect

Team members: Yue Zhang, Xinyu Cheng, Xia Li.

Grading Template
I'm using the following template to grade. Each slot is 10 points. 0 = Missing, 5=OK, 10=Wow!

 09 Executive Summary 09 Installation Instructions 09 User Instructions 10 Highlights (Nice video) 09 Theory of Operation 09 Work Breakdown 09 Future Work 09 Conclusions 10 Demo 10 On time Comments: Looks like a fun project. It's always harder to hook two things together than it looks. Good job.

Score: 93/100

Executive Summary
I believe that almost every college students have played a game called "Greedy Snake" in our childhood. At that time, we played it on the cellphone, on the computer or on the PSP. However, we are always using the keyboard to control the games we played. Now, our final project could bring you something new, that is a "Gesture Controlled Greedy Snake". It means that, using the gesture, the keyboard is never required to play the game. Amazing? Read the following wiki to know how we make it happens!

Basically, we are using the Kinect from Microsoft and the Beagle Board XM from Texas Instrument to accomplish our gesture game. The program for the game is programmed on the QT-Linux. Besides, we use the cross compiling to compile the program to make the game run on the beagle board.

After our previous work, our "Gesture Greedy Snake" is completed, and works pretty good. The snake can be controlled by our gestures smoothly. The game reminds us a different style of childhood memory.

Hardware
1 x Kinect

1 x BeagleBoard xM

1 x DVI to HDMI wire

1 x Monitor

1 x Keyboard

1 x Mouse

Overview
The source files for the project can be downloaded from https://github.com/Astroricks/Snake-Game-with-Kinect-on-BeagleBoard-xM. Once you have set up the developing environment as in the following instructions, you can download the files, playing with the game or making changes.

We are using the drivers provided by Omek. The driver files can be found at the Beckon™ SDK 2.4 – BeagleBoard-xM Edition installation page, and a developer's guide is provided at the documentation page. (You may need to register first.) Instructions in this blog are also very useful.

Although the developer's guide introduces how to do the installation and configuration, we have found that it takes much longer than we originally thought, and errors came out occasionally. It seems that the developer's guide is not very well organized and not detailed enought anyway. So we arranged our instruction and configuration procedures as below:

1. Download the image package and burn it into the SD card
Download the image package Beckon-Image-2.4.19872.tar.gz as mentioned above onto your Linux PC. Then extract it by entering

host$ tar -xvzf 

Run the following command to make sure you have bash installed.

host$ sudo apt-get install bash

Insert your SD card using a card reader/writer. Run the following command:

host$ sudo bash mkcard.sh /dev/ beagle-omek

''Note: This action will wipe out and rewrite the SD card so make sure the files in it are backed up. The name of the SD card can be viewed in Disk Utility. If you mistakenly use the main OS device name this script can erase your entire OS.''

2. Run the sample application
Plug the sensor into the BeagleBoard-xM USB port. Ensure the mouse and keyboard are connected to the BeagleBoard-xM USB port. Connect the BeagleBoard-xM to the monitor with an HDMI cable. Connect the BeagleBoard-xM power supply and wait for the boot process to reach the login prompt, login as user root (no password is required). Now it's ready to run the sample applications on the BeagleBoard-xM.

''Note: Details about running the samples can be found in the developer's guide from page 9 to 11. The Kinect will react properly only if the player is within an appropriate distance (about 1m - 5m). ''

Note: The following instruction is about how to set up the environment for developing. If you don't want to do the development, you can just skip that.

3. Setting up your Linux development PC
Download Beckon-SDK-2.4.19872.tar.gz and extract it onto your host PC (this is the actual SDK which contains headers, libraries, documentation, makefiles, etc...) The SDK provides the tools to develop a motion-tracking and gesture-based application that can run on the BeagleBoard-xM. After developing the application, cross-compile it onto the Linux PC and then copy the image onto the BeagleBoard-xM via external USB storage.

1) Download the Linux ARM cross toolchain:

For 32-bit Ubuntu you will want to download this file: http://www.angstrom-distribution.org/toolchains/angstrom-2011.03-i686-linux-armv7a-linux-gnueabi-toolchain-qte-4.6.3.tar.bz2

For 64-bit Ubuntu you will want to download this file: http://www.angstrom-distribution.org/toolchains/angstrom-2011.03-x86_64-linux-armv7a-linux-gnueabi-toolchain-qte-4.6.3.tar.bz2

Extract the downloaded archive into your system root directory:

host$ sudo tar -xvjf  -C /

Note: If you extract the archive into a different directory, you should adjust the following instructions as well as the samples' makefiles accordingly.

2) Install the Qt embedded SDK for Angstrom/BeagleBoard:

Now that you have your cross compiler installed, you can compile Qt with it. First thing to do is to install zlib library:

host$ sudo apt-get install zlib1g-dev

Then download the Open Source LGPL version of Qt libraries 4.6.2 for Embedded Linux here.

''Note: Newest version of Qt libraries can be found here. We didn't test it since the 4.6.2 version works fine.''

Extract the file you just downloaded:

host$ tar -xvzf 

You need to create a new make.conf. Run the following line to make a new mkspecs directory for the BeagleBoard Processor.

host$ cp -R /mkspecs/qws/linux-arm-g++/ /mkspecs/qws/linux-DM3730-g++/

Edit the qmake.conf file found in /mkspecs/qws/linux-DM3730-g++/ and overwrite it with the following lines.


 * 1) qmake configuration for building with arm-linux-g++
 * 1) qmake configuration for building with arm-linux-g++

include(../../common/g++.conf) include(../../common/linux.conf) include(../../common/qws.conf)


 * 1) modifications to g++.conf
 * 2) Toolchain

QMAKE_CFLAGS_RELEASE =  -O3 -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp QMAKE_CXXFLAGS_RELEASE = -O3 -march=armv7-a -mtune=cortex-a8 -mfpu=neon -mfloat-abi=softfp
 * 1) Compiler Flags to take advantage of the ARM architecture

QMAKE_CC = /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/bin/gcc QMAKE_CXX = /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/bin/g++ QMAKE_LINK = /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/bin/g++ QMAKE_LINK_SHLIB = /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/bin/g++

QMAKE_AR = /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/bin/ar cqs QMAKE_OBJCOPY = /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/bin/objcopy QMAKE_STRIP = /usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/bin/strip
 * 1) modifications to linux.conf

load(qt_config)

Now run the configure command (see this website for more details if needed):

host$ cd 

For 32-bit Ubuntu:

host$ ./configure -opensource -confirm-license -prefix /opt/qt-arm -no-qt3support -embedded arm -little-endian -xplatform qws/linux-DM3730-g++ -qtlibinfix E

For 64-bit Ubuntu:

host$ ./configure -opensource -confirm-license -prefix /opt/qt-arm -no-qt3support -embedded arm -little-endian -platform qws/linux-x86_64-g++ -xplatform qws/linux-DM3730-g++ -qtlibinfix E

Once this is complete you can run make and make install (make may take some hours):

host$ make host$ sudo make install

3) Install and set up Qt Creator:

Go to http://qt-project.org/downloads to download Qt. Get the latest version of Qt for Linux/X11; be sure to get the appropriate 32bit or 64bit version. Once downloaded go to the download directory and execute the following using your download file name:

host$ chmod u+x  host$ ./

Update:The free version Qt Creator can not be used in this project since it doesn't have cross-compile option with it.

After Qt Creator is successfully installed, start Qt Creator. Go to: Tools > Options > Build & Run > Tool Chains, select Add -> GCC and insert the full compiler path below:

/usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/bin/g++

Then go to: Tools > Options > Build & Run > Qt Versions, select Add and provide the following full path to qmake

/opt/qt-arm/bin/qmake

Press Apply and OK.

4. Setting up the BeagleBoard-xM
All of the Qt files that you need to move to the BeagleBoard can be found in the /opt/qt-arm/lib directory on your Linux Build System. Copy all the files out of your build system's /opt/qt-arm/lib directory and put those into the BeagleBoard-xM's /opt/qt-arm/lib directory. You can get the files to the Beagleboard by either networking, using a usb drive or plug the SD card into your Linux Build System.

Building and running samples
To build QtTracking Sample:

1. Start Qt Creator.

2. Go to: File > Open File or Project, select the QtTracking-SDK.pro project (from Beckon-SDK-{version}/samples/QtTracking) and click Open. Select Embedded Linux instead of Desktop.

3. Click Finish.

4. Go to Projects tab (on the left), In Qt version select Qt [4.x.x (qt-arm)]/.

5. Run Build -> Build Project command.

The result executable can be found in the Beckon-SDK-{version}/bin directory.

To run the QtTracking Sample:

1. Mount or copy the bin directory to the BeagleBoard.

Note: Run the reset.sh script inside the "bin" folder of the SDK before each sample run, to reset the DSP.

Run the following:

beagle$ ./QtTracking-SDK –qws

Note: Developer's guide has more instructions on running the sample

The Snake Game
The Snake Game can be downloaded from https://github.com/Astroricks/Snake-Game-with-Kinect-on-BeagleBoard-xM. To play with it you should copy the folder to the root directory of SD card. Since what we did is combining the Snake Game source code into the Tracking Sample, running this application would be the same way as running the samples.

We suggest playing with the following command:

beagle$ cd Tracking_Sanke/bin/ beagle$ ./QtTracking-SDK –qws -upper

When the application is running, we suggest pressing the "setting" button first to enable the wall-cross mode, which could make it not so easily "game over". To start the game, you need to press Up Arrow(or Arrows of other directions) on the keyboard. And the player would get a better playing experience if he/she stands in front of the Kinect with a distance of about 1.5m - 2m. The player should control the snake with his right hand moving up and down, left and right.

Highlights
Our project is about how to use Microsoft Kinect Sensor to play snake game by the motion of you right hand(You can modify the codes to change to any part of your body if you would like to). We set a score section to record the highest score. Anybody who want to break our record are welcome(Currently the record is holding by Xinyu Cheng at 260). We've created a play demo and uploaded it to YouTube:

The Main Idea
We are using the kinect as the sensor of our project. The provided algorithm of the kinect make it possible for us to get the coordinate of many joints of the body. In this case, we can judge the position and the motion of our hands or any other joints. Then, we can use the direction we get from the previous judgement to control the Greedy Snake. Our project is running with the idea.

The Technics Used
Technically, we can use QT to design our program and then cross-complie our project to the beagle-board. The Greedy Snake is an open-source QT game, and the tracking of the joint can be accomplished from modifying the Omek SDK. We have to modify both of their codes to connect the two parts together.

The Control Strategy
Judging gestures simply from the changing of coordinate is hard and may lead to mistakes commonly. To improve the playing experience, we combined two methods to guess which control direction the player wants to make.

1. Compare the current coordinate with the last time. When the date is refreshed, we can compare the current coordinate of player's right-hand withe the last time value. If the difference between the coordinates of x or y is larger than the threshold, it is reasonable to think that the player may want the snake to make a move in this direction.

2. See where the hand locates. After making a upward movement, it has a high possibility that the y coordinate locates in the up area. Similar situation appears in other movement directions. So by checking the current value of the x and y coordinates we can know which direction the player want to make move.

Only when the same direction is implied by both methods, we move the snake in this direction. It seems that this control strategy works pretty well.

Yue Zhang

 * Researched how to Burn the correct Image to the Beagle Board.
 * Configure the Beagle Board to control the kinect.
 * Implemented the motion based algorithm which makes the application runs more smoothly.
 * Programmed for the part of motion algorithm.

Xinyu Cheng

 * Researched how to configure the QT programming environment.
 * Accomplished the cross-compiling on QT.
 * Programming for the connection between the Snake Game and the Kinect SDK.
 * Designed and Implemented the coordinate based algorithm.

Xia Li

 * Researched how to run the provided SDK demo.
 * Speed up for the demo.
 * Designed the motion based algorithm.
 * Programmed to improved the accuracy and speed of the game.

Future Work
Now, our game can work pretty good and smoothly. The future work for the project will mainly about the following aspects:
 * Improve the user interface, make the game looks more pretty.
 * Improve the game rules, to make it more interesting.
 * Improve our algorithm for the direction judgement, to make the game runs more smoothly.
 * Display the motion of our hand on the window, to make the user more clear about the motion of their hands.

Conclusions
Conculsionly, our project works pretty good. Our Greedy Snake game runs very good and brings a lot of fun for us.

During the develop process, we met lots of difficulties, we tried to solve them one by one. The first problem we met is how to build the connection between the kinect and the game. We tried very hard to debug our program step by step, then finally found the bug and fixed it.

We can still do something more to improve our project. Like we have talked about in the "Future Work" part, we can try to improve the GUI of our game to make it looks more pretty, and improve our display inforamtion and the algorithm for the game to make it easier for users to play. So, to make the game perfect, there is still a long way to go.