ECE497 Project: Kinect Project
Our goal is to implant a specific game application into Beagleboard and using Microsoft's Kinect to play with it. We would firstly make the Kinect recognizable to the Beagleboard. And then try to adapt the application to the Beagleboard. We haven't decided which game to play with, because the performance may be greatly influenced by what kind of operating system we want to use, or the size of this game. So the second part would be to test what is the best option for us. Finally, we would try everything to make it work.
Update: We finally decided to use Omek Beckon 2.4 SDK on BeagleBoard as interface of Kinect to develop our application and we chose Qt creator as cross-compiler tool to program and debug. As the limitation of the image(we could only use the image Omek provided with us, see Omek Forum for details, you may need to register first), we can't transfer all the drivers and data to the image we used to use and also we can't develop programs with GUI using Qt, because the Omek image only support command-line mode, so we chose to develop the GUI itself within the source code. We searched on line and found the open source code for a game called Q-ball. We are going to use kinect to play with it.
1. Burn the Image Package and Boot the System
The detailed instruction is shown in page7 of developer's guide from omek.
When I tried to run mkcard.sh to create new partitions, I got some errors.
One of the error is the shell script is asumming that first partition of device "mmc" is "mmc1" but it's "mmcp1" in my Ubuntu. I changed the shell script to fix this but still got some errors about the partition specs.
Finally I found a tricky way to use the shell script directly. If I use a external USB card reader to read and write the microsd card, everything just works perfectly. I think maybe the author of shell script had a different verison of Linux or only tested the script on descktop with external card reader.
2. Run the Sample Demos
After burning the microsd card sucessfully, we can boot from this card and see a simplified Angstrom without GUI. All demos ran almost as expected but,
(1).the program froze sometimes. It even happened sometimes when we were just typing some command. We think maybe it's because of the speed of microsd card. We will get a faster card and have a try.
(2).the tracking for legs doesn't work very well, we think maybe it's because we didn't have enough space for the kinect.(When you played kinect game in xbox, it will tell you a large space is required for tracking).
3. Compile the Sample Code on Host
Detailed instruction is provided in page26 of the developer's guide from omek. By following the instruction, we downloaded and installed the Linux ARM cross toolchain. We also installed zlib library and cross compiler for qt. But we had some problem when compiling qt. The error information says the make cannot find the cross-compiler. I used export command to set the path of cross complier we used in class before(some time later I noticed this may cause some problem) and new error occured saying "undefined reference to clock_gettime". I googled it and find it's involved in one library so I modifed the make file and include the library when compiling but it still gave the same error.
Then I went through the instruction carefully and found actually the instruction asked us to install two different toolchains for the same purpose. The difference of two toolchains is the version of qt, one is 4.6.3 and the other is 4.6.2, since our souce code is for 4.6.2 version, so I want to try toolchain for 4.6.2 to see if it makes any difference.
The instruction also asked us to install two different versions of QtCreater in the pdf file and in blog, we prefered the version in pdf file since the instruction should be written based on this version.
After struggling for a very long time, I decided to clean everything up and start over. I followed the instruction one step by step but skipped one step and it turned out everything works perfectly now. Link Omek Forum may help you with the version problem.
Some conclusion for compiling Qt:
(1). I reinstalled the Ubuntu when started over. It turned out you need to install g++ compiler before before getting started.(Type "sudo apt- get install g++"). Other than that, you don't need to do anything not mentioned in the developer's guide and the blog it quotes.
(2). The instruction from Omek is a little redundant with the blog it quotes and not taltally right.(At least they don't all work on my laptop)
(3). The "developer's guide" pdf file and the blog it quotes both tell developer to install a toolchain( but different version, it's 4.6.3 in the pdf file and 4.6.2 in the blog) for cross-compiling. The 4.6.3 toolchain works perfectly for compiling Qt 4.6.2.
(4). You don't have to install QtCreater before compiling Qt. The blog says you'd better download one Qtcreater from the nokia website so that you can get the latest version, but it's not really necessary according to my experience. You can get a pretty good and working version by simply running "sudo apt-get install qtcreator".
(5). In the instruction for building QtTracking Sample, step 3.b tells you to remove the existing qmake step and make you own custom step, but I always got error when the building process reached the costum step, so I tried to just leave the existing qmake there, and everything worked amazingly!!! So, if you get trouble about the costum step you made in building, you can just try to use the default qmake("existing qmake" in the instruction from Omek).
4. Build Q-ball without adding Kinect support using Qtcreator After setting up ready all the development environment, we are going to firstly build the game without Kinect to see if it can work. The source code is already Qt coded, so we didn't meat any problems here. The code could successfully run on our beagleboard.
5. Build Q-ball with Kinect support using Qtcreator Lastly and most importantly, we would add Kinect support to Q-ball. The source code is written in C++, but both of us do not have any experience in it. The process of programming and debugging is quite struggling. We could finally make it work, the BeagleBoard could recognize our left and right scroll gestures, but the speed is very slow. We may proceed with this, and make it faster if more time is allowable.
- Github links: Github, including Omek image, developer guide, Omek sample code, and Kinect supported Q-ball source code
- Omek Beckon SDK Development tools
1. If you haven't installed git, see instrucstions here EBC Exercise 07, if you did, git pull our reposotiry to your host.
host$ git clone firstname.lastname@example.org:wangg/ECE497.git host$ cd ECE497/
2. Insert your SD card, we recommend you to use an external USB card-reader, which would avoid strange mistakes.
sudo bash mkcard.sh /media/sdX beagle-omek
WARNING: If you mistakenly use the main OS device name this script can erase your entire OS.
3. Connect your BeagleBoard, insert your newly made SD-card, plug in the power supply, Kinect, mouse and keyboard. The log in user is root without a password.
4. Now we want to run the demo program Omek gave us to see if everything works.
NOTICE: The code has been modified for Q-ball, you need to untar Beckon-SDK-2.4.16236.tar.gz file first in order to run the sample.
To run Qt Tracking Sample in live tracking using camera input:
# sh ./tracking.sh
or if you can record a camera sequence using the tools provided with the PC version of OpenNI. To run prerecorded camera sequence (OpenNI sequence file (.oni)):
# sh ./tracking.sh -seq path/to/directory/containing/the/sequence/file/
Note: the path points to a directory containing the sequence, not to the sequence file itself.
To run TrackingViewer3D Sample in live tracking using camera input:
# sh ./tracking3d.sh -gest <gesture name> -gest ...
To run GestureDemo Sample, you need to run
# sh ./gestures.sh -gest <gesture name> -gest ...
The supported list of gestures are: • _rightClick _leftClick • _rightScrollRight _rightScrollLeft • _rightScrollUp _rightScrollDown • _leftScrollRight _leftScrollLeft • _leftScrollUp _leftScrollDown
Note: The path points to a directory containing the sequence, not to the sequence file itself.
5. Now we are moving to the host side, and install the toolchain and Qtcreator we need.
Theory of Operation
1. Installed the Beckon SDK Development tools on both Linux PC host and BeagleBoard; Successfully tested the sample code and discussed the interface functions. ---completed on February 5th by Yifei Li and Guanqun Wang