ECE497 Project: PS EYE QT

Team members: Mona Yan, Greg Harrison

Grading

Greg

Your part of the wiki is very complete. Nicely documented. Cleaver workaround to get the plots running quickly enough. Good Job.I'm able to do the git pull and run everything as described. The only problem is the waveforms that are displayed don't seem to correlate with the input sound. I whistle and expect to see some sort of sine wave, but I don't. Your code could use some more documentation. Did you create it all on your own, or did you find examples elsewhere? Your comments should say.

Overall very good, but lacking some functionality.

Wiki:     100
Project:  85
Total:    92.5

Mona

Your wiki description could have more details. What do the different options of your program do? How do I quit? I don't hear anything coming out. How do I switch to the PS EYE microphones?

Wiki:     90
Project:  80
Total:    85

Executive Summary

For our Final Project we wanted to be able to capture audio from the PlayStation Eye microphone array and then be able to display the audio waveforms all on the beagle. Initially we set out to display the waveforms in Qt using Qwt, which was later proved to be more difficult than initiall anticipated due to limitations on the Beagle and neither of us having prior C++ programming experience. The next thought was to try to use GnuPlot to try to plot the audio data. GnuPlot was really designed to produce static plots of information, not plots that were in realtime. We were able however to find different ways to "force" GnuPlot to plot the live data with the help of a clever script written by Thanassis Tsiodras, and then also edited by Andreas Bernauer which can be found here. When we implemented this script with our audio data on the beagle it still wasn't as realtime as we would have liked it to be due to the capabilities of the beagle, so we then had the idea to send the data back to a host computer which would then have the processing capability to produce the results we were after.

We currently have working plots of audio waveforms for channel1 and channel2 of the PS Eye microphone array. These plots are displayed on a host computer with the data being sent from the beagle via netcat. If time permits, it would be interesting to see if what we could also get netcat to do such as transferring video and/or audio from the beagle to the host.

We've investigated the audio effect through open source community and we choose Gstreamer. We explored GStreamer core module, the gst-plugins-base module, the gst-plugins-good module, the gst-plugins-ugly module, and the gst-plugins-bad module offered by the 0.10 GStreamer stable release series. We successfully implemented lowpass, bandpass filter, echo, reverberation and invert effects. In addition, we further explored other audio effects that Gstreamer capable of, such as removing human voice from background music.

This project turned out to be a lot more difficult then initially anticipated. Although it would be nice for this project to run solely on the beagle it is very interesting to discover all of the possibilities of utilizing netcat between the beagle and host.

Installation Instructions

• This is the github path to our project: https://github.com/harrisgw/ECE497-DisplayAudioWaveforms .

Every package needed on the beagle should already be there with the current image we are working with. Start by cloning the git directory on the host computer:

user@host:~$ git clone git@github.com:harrisgw/ECE497-DisplayAudioWaveforms.git

Then go into this directory and view all the files

user@host:~$cd ECE497-DisplayAudioWaveforms
user@host:~/ECE497-DisplayAudioWaveforms$ ls
Beagle  driveGnuPlotStreams.pl  Host.sh  README

The folder Beagle now needs to be copied to the beagle. In this example 10.0.0.13 is the IP address of the beagle replace with your beagles IP address:

user@host:~/ECE497-DisplayAudioWaveforms$ scp -r Beagle/ root@10.0.0.13:.
root@10.0.0.13's password: 
audio_input_output.h                            100%  521     0.5KB/s   00:00    
Makefile                                        100% 6923     6.8KB/s   00:00    
Beagle.sh                                       100%   56     0.1KB/s   00:00    
audio_input_output.c                            100%   12KB  11.7KB/s   00:00    
debug.h                                         100%  741     0.7KB/s   00:00    
audio_thread.h                                  100%  447     0.4KB/s   00:00    
audio_process.c                                 100%  879     0.9KB/s   00:00    
audio_process.h                                 100%   98     0.1KB/s   00:00    
app_DEBUG.Beagle                                100%   23KB  23.4KB/s   00:00    
main.c                                          100% 1806     1.8KB/s   00:00    
audio_thread.c                                  100% 9553     9.3KB/s   00:00 

The Beagle folder should now be on your beagleboard. Check and then enter that folder:

root@beagleboard:~# ls
Beagle	 a.out	   busybox	driveGnuPlotStreams.pl	exercises  sinuses.pl
Desktop  autogain  byobu-4.25	esc-gst			network
Old	 bin	   c6run_build	esc-media		qt
root@beagleboard:~# cd Beagle
root@beagleboard:~/Beagle# ls
Beagle.sh	  audio_input_output.c	audio_process.h  debug.h
Makefile	  audio_input_output.h	audio_thread.c	 main.c
app_DEBUG.Beagle  audio_process.c	audio_thread.h

Now that that is all set up we need to get the gnuplot package on the host. This should be all you need to do:

user@host:~/ECE497-DisplayAudioWaveforms$ sudo apt-get install gnuplot   (took 33 seconds)

For the audio effect, compile AudioEffect.c to read in audio from in input jack.

# gcc -Wall $(pkg-config --cflags --libs gstreamer-0.10) AudioEffect.c -o AudioEffect

Run AudioEffect.

# ./AudioEffect
usage: please input <choice> (choice from 0 to 4)
0--original
1--lowpass
2--bandpass
3--audioecho-reverb
4--audioinvert

User Instructions

After setting up everything on the host and Beagle, its time to see if everything works. First run the Beagle.sh script on the Beagle board:

root@beagleboard:~/Beagle# ./Beagle.sh

Then on the host run the Host.sh script, which will ask a series of questions:

user@host:~/ECE497-DisplayAudioWaveforms$ ./Host.sh 
How many streams (enter:1 or 2)?
2
Would you like those streams in 1 window or 2 windows? (enter:1 or 2)
2
How many samples per window?
100
Enter Minimum Y axis:
-10000
Enter Maximum Y axis:
10000
Enter window width:
500
Enter window height:
500
Enter IP address of Beagle:
10.0.0.13

If everything worked you should now see 2 plots of audio that are from the Beagle board being displayed on the host computer. To quit the plots just push ctrl-c in the terminal window of either the beagle or the host.

As to the AudioEffects part, it reads audio data using alsascr. Input some sound source to the aux-in, and connect the aux-out with earphones or speaker with left and right channels.

When you run the AudioEffects, several options will pop out:

usage: please input <choice> (choice from 0 to 4)
0--original
1--lowpass
2--bandpass
3--audioecho-reverb
4--audioinvert

Input different choice, it give sound after low-pass and bandpass filters. You can also hear the echo with reverberation and an invert effects. In addition, the variables of filters can be modified in the AudioEffects.c file, according to customized effects.

Highlights

Currently our project can take audio data from a Playstation Eye microphone array running on the Beagle Board and then plot it on a host computer. We found this method to be the fastest and smoothest way to implement the plotting of the audio signals with the limited C++ programming experience and time we had. The script running on the host can take information about the plots including the range, number of channels, number of windows, and the number of samples displayed. A demo of what our project can do can be found here.

Theory of Operation

Our program works by running the AudioThru program we made work in Lab08 to generate the audio data. This audio data is then sent through a netcat connection with the host computer. This audio data is then plotted using GnuPlot with the given preferences for the plots supplied by the user. Scripts were created to simplify this entire process. The plots can be generated by only running one script on the beagle and one script on the host computer making the plotting easier for the user. The simple overview of our program can be summed up in the following diagram:

GStreamer Pipeline

It reads audio data using alsascr. After applies sound effect, it outputs the audio using alsasink. The Gstreamer elements we used for our audio effects include audioconvert, audiocheblimit, audiochebband, audioecho, audioinvert, etc. For detailed information please refer to GStreamer.

Work Breakdown

Initial work with Qt-Greg Mona

Plotting with gnuplot-Greg

Investigating open source media framework- Mona

Sending data back to host for faster plotting with netcat- Greg

Making scripts to somewhat automate the plotting- Greg

Audio effect using Gstreamer- Mona

Conclusions

We are able to capture channel1 and channel2 audio signal from PS Eye microphone and display audio waveform plot on a host computer with the data being sent from the beagle via netcat. We also able to implement four different audio effects using Gstreamer. But this project turned out to be a lot harder than first anticipated. Much time was spent trying to get a Qt application to run on the beagle and use Qwt to plot the audio data with not a lot of success. It would be nice if we can add GUI to audio effect, then user can choose the effect and set properties of each effects. It would definitely have been better if we had C++ experience and a familiarity with Qt. It would be very beneficial if time was not a factor to learn how to use Qt because it can provide some powerful applications especially on embedded devices. Though we couldn't get this project to work using Qt and to run soley on the beagle, we did want to have something to show for our project which is the current implementation. Getting a program to run only using the beagle, adding a GUI to enhance the user experience with our program, and having the ability to send video and audio back to a host computer are all enhancements that would be greaat to implement if time permitted.