View source for ECE497 Project Chronos Controlled Audio Effects

[[Category:ECE497Spring2011]]

Team members: [[user:carlsojs|Joel Carlson]], [[user:hulettbh|Brian Hulette]]

== Executive Summary ==
With this project, the BeagleBoard is given an audio input, applies various effects to that audio, and then outputs the audio.  A [http://processors.wiki.ti.com/index.php/EZ430-Chronos?DCMP=Chronos&HQS=Other+OT+chronoswiki Texas Instruments Chronos watch] is used to control which effects are applied, and the watch's accelerometer data is used to adjust the parameters of the effects.

At this point, the project is working with three effects.  The three effects are a normal pass-thru, a lowpass filter, and a bandpass filter.  The watch's accelerometer data controls the cutoff frequency for the lowpass filter, and it controls the center frequency and bandwidth for the bandpass filter.  GStreamer is used to provide the audio effects.

At this point, the only things "not working" would be that we did not implement the highpass filter, reverb, or flanging that we initially planned on including.  These would be fairly easy to integrate into the program as long as one can implement it with GStreamer.

Further information about GStreamer and the TI Chronos Watch can be found below.

== Installation Instructions ==
=== Git Repository ===
The project is stored on GitHub: https://JoelsonCarl@github.com/JoelsonCarl/Chronos-BeagleBoard-Audio

Other URLs:<br />
Read+Write Access: git@github.com:JoelsonCarl/Chronos-BeagleBoard-Audio.git<br />
Read+Write Access: <nowiki>https://JoelsonCarl@github.com/JoelsonCarl/Chronos-BeagleBoard-Audio.git</nowiki><br />
Read Only Access: git://github.com/JoelsonCarl/Chronos-BeagleBoard-Audio.git

=== Compilation and Installation ===
The program was compiled on a host machine running Ubuntu 10.04 64-bit.  The program was cross-compiled using the [http://www.angstrom-distribution.org/toolchains/angstrom-2011.03-x86_64-linux-armv7a-linux-gnueabi-toolchain-qte-4.6.3.tar.bz2 angstrom-2011.03-x86_64-linux-armv7a-linux-gnueabi-toolchain-qte-4.6.3] toolchain.  To compile, the <code>CC_ROOT</code> variable within the file '''makefile_profile.mak''' must be changed to point to <code>%TOOLCHAIN_PATH%/usr/local/angstrom/arm</code>.

The <code>install</code> target within '''makefile_profile.mak''' may need to be changed.  We had it <code>scp</code> the executables to <code>/home/root</code> on the BeagleBoard.  Essentially, whether or not you use <code>make install</code>, you need some way to get the '''chronos-beagle_DEBUG.beagle''' and/or '''chronos-beagle_RELEASE.beagle''' file to the BeagleBoard.

=== Additional BeagleBoard Packages ===
The following additional packages were needed from those already on the SPEd2 image.
* none

=== Kernel Modifications ===
'''The kernel included with the SPEd2 image does not require these modifications.'''

(The kernel we used is included in the git repository as '''uImage-chronos''')

The kernel we used is the version pulled via the OpenEmbedded bitbake process (linux-omap-psp-2.6.32-r100a+gitr5fc29e7b2a76a64a739f857858ef0b98294aa155).  After we had that kernel, we did the following.

Clean up the kernel, create a config file with the default settings, and then make the default kernel.
<pre>
$ make mrproper
$ make omap3_beagle_defconfig
$ make
</pre>

After the kernel has compiled, run <code>make menuconfig</code>.  In the configuration options, go into <code>Device Drivers -> USB Support</code>.  Within there, put a "Y" on the following:
* USB Modem (CDC ACM) Support
* USB Serial Converter Support
** USB TI 3410/5052 Serial Driver
Exit menuconfig and save the changes.

Compile these changes and then build the uImage to boot from.
<pre>
$ make
$ make uImage
</pre>

=== Extra Hardware ===
The TI Chronos watch and RF access point can be purchased from [http://focus.ti.com/docs/toolsw/folders/print/ez430-chronos.html?DCMP=Chronos&HQS=Other+OT+chronos http://focus.ti.com/docs/toolsw/folders/print/ez430-chronos.html?DCMP=Chronos&HQS=Other+OT+chronos].

== User Instructions ==
# Kill any running PulseAudio processes. (Run <code>killall -9 pulseaudio</code>)
# Run the '''chronos-beagle_RELEASE.beagle''' file.  The output should say <code>Starting Effect (0): Pass Thru</code>.
# On the Chronos watch use the bottom left (M2) button to navigate between various watch modes.  Go to either '''ACC''' or '''PPT''' mode.
#* PPT Mode
#** Press the bottom right (S2) button to enable PPT mode and connect to the RF access point (a wireless symbol should start flashing on the watch).
#*** Pressing the top right (S1) button will start or stop the audio.
#*** Pressing the top left (M1) button will switch between different effects.
#**** 0 = Pass Thru
#**** 1 = Lowpass Filter
#**** 2 = Bandpass Filter
#*** Pressing the bottom left (M2) button currently has no functionality.
#*** Pressing S2 again will stop PPT mode.
#* ACC Mode
#** Press the S2 button to enable ACC mode and connect to the RF access point (a wireless symbol should start flashing on the watch).
#*** Pass Thru - the accelerometer has no effect
#*** Lowpass - tilting the watch away from you lowers the cutoff frequency, tilting towards you increases the cutoff frequency
#*** Bandpass - tilting the watch away lowers the center frequency, tilting towards increases the center frequency, tilting left narrows the bandwidth, tilting right widens the bandwidth
#** Press the S2 button again to stop ACC mode.
# Press Ctrl-C to stop the program.

== Highlights ==

We have nothing to highlight or brag about beyond what was already described.

== Theory of Operation ==

=== Chronos Watch ===
The chronos.c file contains the "main" running loop.  This program opens up the serial port to interact with the RF access point.  It then enters a loop where it repeatedly writes a command to the serial port that will request data from the watch.  It then waits for that data to be returned.  If the data is a button press or accelerometer data, the program will send the appropriate commands to the GStreamer pipeline.  The relationship between watch events and GStreamer commands is consistent with that described in the [[#User Instructions | User Instructions]].  After those commands are sent (or if no button press or accelerometer data was supplied), the program sends out another request for data and repeats the process over and over until an interrupt (Ctrl-C) is seen.

=== GStreamer Pipeline ===
The pipe.c file contains an implementation of an audio pipeline made with [http://gstreamer.freedesktop.org/ GStreamer].  GStreamer is a tool that allows you to connect various blocks (called elements) together to create a pipeline which multimedia data (in our case, just audio) flows through.  Our pipeline is actually quite simple, it reads audio data using ALSA, then applies some effect to it (using one of the many built-in audio processing plugins), and finally outputs the audio using ALSA:

<pre>
alsa-src -> audioconvert -> EFFECT -> audioconvert -> alsa-sink
</pre>

The EFFECT element can be any number of GStreamer plug-ins.  The three that are currently implemented are:
{| border="1" cellspacing="0" cellpadding="2"
!scope="col"| Name
!scope="col"| Function
!scope="col"| Sample Pipeline
|-
| [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-audioconvert.html audioconvert]
| Pass-thru
| gst-launch alsasrc ! audioconvert ! audiconvert ! audioconvert ! alsasink
|-
| [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-audiocheblimit.html audiocheblimit]
| Low-Pass Filter
| gst-launch alsasrc ! audioconvert ! audiocheblimit mode=low-pass cutoff=800 ! audioconvert ! alsasink
|-
| [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-audiochebband.html audiochebband]
| Band-Pass Filter
| gst-launch alsasrc ! audioconvert ! audiochebband mode=band-pass upper-frequency=1500 lower-frequency=1000 ! audioconvert ! alsasink
|}

The sample pipelines show an example of how we are using each filter.  You can run each of these at the command line on a computer with gstreamer-0.10 installed to get an idea of what each effect does.

The creator of the pipeline has the ability to:
* Start the pipeline
* Stop the pipeline
* Restart the pipeline with a different effect block
* Edit the effect parameters while the pipeline is running

== Work Breakdown ==
{| border="1" cellspacing="0" cellpadding="2"
!scope="col"| Task
!scope="col"| Implementor
|-
| Figure out how to interface with Chronos RF access point and get watch data
| Joel C.
|-
| Look into ways to implement audio effects
| Brian H.
|-
| Implement audio effects
| Brian H.
|-
| Merge Chronos and audio function calls
| Joel C.
|}
For the audio effects, Brian looked into [http://focus.ti.com/docs/toolsw/folders/print/sprc265.html DSPLIB], [http://gstreamer.freedesktop.org/ GStreamer], and [http://uazu.net/fidlib/ fidlib].  He ended up using GStreamer in the project.

Only a pass-thru, lowpass, and bandpass audio effect were implemented.  Further effects (using GStreamer) could be implemented very easily.

== Conclusions ==
The program is fully functional.  Other than the watch occasionally registering a single button press as multiple presses, it is generally an easily usable program.  The switching between audio effects happens quickly and smoothly enough that it is not too unpleasant to listen to.

Figuring out how to interface with the Chronos RF access point/watch was not easy.  The example programs we found were useful, but it was still not the same as a good set of documentation on what commands the RF access point expected as well as how the watch would return the data.  We also had a lot of difficulty finding and linking a library for running audio filters.  As mentioned earlier, we investigated several different alternatives for the audio processing.  In the process of trying to get these alternatives working we learned a lot about cross-compiling and linking cross-compiled libraries.

Future additions could include adding more audio effects using GStreamer.  Also, the bottom left watch button (M2) did not end up getting used, so someone could probably come up with a good use for that.

== Extra Information ==
=== TI Chronos Watch ===
We are using the default watch firmware.

We found a Python script that pulls button presses (found [http://chemicaloliver.net/programming/receiving-ti-ez430-chronos-button-presses-in-processing-and-python/ here]).  We placed the original Python script in the git repository.  We used this as a start for writing C code to get the button presses.

==== Chronos RF Access Point Commands ====
The RF access point takes certain commands over the serial port to communicate with the watch.  Some of these commands are listed below.  These were found [http://e2e.ti.com/support/microcontrollers/msp43016-bit_ultra-low_power_mcus/f/166/t/32714.aspx here].

All values listed are hexadecimal.
* Background Polling (functionality unknown)
** PC to AP: FF 20 07 00 00 00 00
** Response: FF 06 07 xx xx xx xx
** xx xx xx xx: 4 byte watch address
* Start Access Point
** PC to AP: FF 07 03
** Response: FF 06 03
* Check AP Command (functionality unknown)
** PC to AP: FF 00 04 00
** Response: FF 06 04 03
** It may be returning the state of the watch or AP
* Request Data
** PC to AP: FF 08 07 00 00 00 00
** Response: FF 06 07 tt xx yy zz
*** tt: data type
**** 12 (M1 pressed)(xx,yy,zz = 00)
**** 22 (M2 pressed)(xx,yy,zz = 00)
**** 32 (S1 pressed)(xx,yy,zz = 00)
**** 01 (valid acc data)(xx,yy,zz = acc data)
**** FF (no/invalid data)
* Stop Access Point
** PC to AP: FF 09 03

=== Cross-Compiling with GStreamer ===
Cross-compiling code that uses GStreamer can be somewhat of a pain.  The difficulty arises because you must find libraries that have been compiled for the ARM, and not for your host machine.  After looking at some GStreamer documentation I found that compiling on the beagle is possible using the command:
<pre>
$ gcc -Wall $(pkg-config --cflags --libs gstreamer-0.10) pipe.c -o pipe
</pre>
The <code>pkg-config</code> portion of this command is replaced with the c-flags necessary to link GStreamer when it is run.  Unfortunately, running this command on your host machine causes it to point to libraries compiled for your host, not the ARM.  I had to figure out what C flags are actually being generated here so that I could modify them to point to the appropriate ARM libraries.  Running the <code>pkg-config</code> command by itself yields:

<pre>
$ pkg-config --cflags --libs gstreamer-0.10
-pthread 
-I$/usr/include/gstreamer-0.10 
-I$/usr/include/glib-2.0 
-I$/usr/lib/glib-2.0/include 
-I$/usr/include/libxml2 
-pthread -lgstreamer-0.10 -lgobject-2.0 -lgmodule-2.0 -lxml2 -lgthread-2.0 -lrt -lglib-2.0
<<< Formatting added by author for clarification >>>
</pre>

We can use all of this as-is except for the include flags.  Versions of these libraries compiled for the ARM can be found deep in the toolchains directory.  On my machine they are here: <code>/home/hulettbh/Beagleboard/toolchains/usr/local/angstrom/arm/arm-angstrom-linux-gnueabi/usr/</code>.  So, in order to create a Makefile which properly compiles with GStreamer you must create the c-flags like so (with the directories modified slightly for your machine, of course):
<pre>
ARM_TOOLCHAIN_PATH  := /home/hulettbh/Beagleboard/toolchains/usr/local/angstrom/arm

ARM_INCLUDE	    := $(ARM_TOOLCHAIN_PATH)/arm-angstrom-linux-gnueabi/usr/include
ARM_LIB		    := $(ARM_TOOLCHAIN_PATH)/arm-angstrom-linux-gnueabi/usr/lib

ARM_CFLAGS += 	-Wall -pthread \
		-I$(ARM_INCLUDE)/gstreamer-0.10 \
		-I$(ARM_INCLUDE)/glib-2.0 \
		-I$(ARM_LIB)/glib-2.0/include \
		-I$(ARM_INCLUDE)/libxml2 \
		-pthread -lgstreamer-0.10 -lgobject-2.0 -lgmodule-2.0 \
		-lxml2 -lgthread-2.0 -lrt -lglib-2.0
</pre>