BeagleBoard/GSoC/2021 Proposal/TensorFlow Lite Compatibility with BeagleBone AI
TensorFlow Lite Compatibility with BeagleBone AI
About
Student: Leah Pillsbury
Mentors: Contacted Robert C Nelson
Code: TensorFlow Lite Support
Wiki: Proposal/TensorFlow Lite Compatibility with BeagleBone AI
GSoC: N/A
Status
This project is currently just a proposal.
About you
IRC: lpillsbury
Github: lpillsbury
School: Pasadena City College (Boston University prior to PCC)
Country: United States
Primary language : English
Other languages : Spanish, Swahili, Hebrew, some Hindi/Urdu, Telugu, Bengali
Typical work hours : 9AM-5:30PM US Pacific, though I may be on Eastern time part of the summer
Previous GSoC participation: First time participation in GSOC. I am interested in learning more about embedded software, and participating in the community. I use open source tools regularly, and it is exciting to make something useful that other people would use too.
About your project
Project name: TensorFlow Lite Compatability with BeagleBone AI
Description
Currently, TensorFlow Lite is supported by Raspberry Pi, Arduino, ESP32, multiple Adafruit Boards, Android, and iOS, among others. No BeagleBoard is on this list, and it should be. The specs of BeagleBone AI are impressive, and it does allow for other AI platforms such as Caffe, it seems limiting for an AI board for the general public to not be able to access TensorFlow Lite. I would like to change that by choosing a current release of TensorFlow Lite, and writing the code necessary to run it on the Arm M4 processors that the BeagleBone AI board already has.
This project will be a combination of C coding, dealing with the Linux kernel, and doing some example cases with TensorFlow Lite in python. Obviously, given the diversion between OpenCL and TensorFlow, there is no perfect way to seamlessly integrate TensorFlow Lite, but there seem to be enough people in the community who want it, that it is worth at least making a gesture towards a compatibility solution. I first realized this was a problem when I decided I wanted to learn more about AI on the edge. One of the first things that comes up when I search for information on this topic is using TensorFlow Lite and the wide variety of other boards that it plays well with.
Ideally, I'd spend the first few weeks getting TensorFlow Lite working on a BBAI, document my hacking process, and then create a smooth stable way to make it work more out of the box. Then I'd spend the rest of the time doing examples that would be highlighted both on the BeagleBoard website, and also on the TensorFlow Lite Examples page by making a pull request to the owners of the github repo. Given that there are already examples of using TensorFlow Lite on Raspberry Pi, with picamera, the starting point would be to have equivalent BeagleBone examples, most likely doing image capture through OpenCV (OpenCV with BeagleBone Black). I'm also very interested in sound recognition, and I think that recognizing sound patterns is very important for industrial uses.
Getting TensorFlow Lite Working on BBAI
There is an outstanding question as to how hard it will be just to get TensorFlow Lite working on the BeagleBone AI board. I know that:
- Nothing that sounds easy rarely is on an embedded device
- Many people have struggled to get TensorFlow Lite working on BBAI. Some of these have had to switch platforms when they couldn't get it working (See advice to one user in Benefits
- Communication with @jkridner suggests that the issue still needs to be dealt with:
There are bits out there, but rcn-er hasn’t integrated them.# Communication with Terry (Woncheol) Heo at TensorFlow Lite suggests that getting TensorFlow Lite working on BeagleBone AI should be relatively simple, if it is first cross compiled on another machine. He said:If you're talking about the native building (building BeagleBone binary on BeagleBone), you need to make sure you have enough RAM installed. As I said, 1G ram + swap could work but it would be slow. That's why I recommend using cross compilation. I don't think we have a known issue for ARM cross compilation now. FYI, CMake support was added recently. For ARM cross compilation with CMake. you may want to check the following page. https://www.tensorflow.org/lite/guide/build_cmake_arm I think ARMv7 NEON (armhf) binary will work nicely with BeagleBone AI. But please let me know if you have any issues with it.Given this information, I expect that it will take some time to track down what has been tried before and try a few different options to develop a recommended procedure for using TensorFlow Lite on BeagleBone AI. I also don't anticipate it being impossible, given the support for ARM processors and the "bits out there". If it is so easy that I have extra time, I could also try additional tools that haven't seen much testing such as PyTorch, mlpack etc.AI Examples Using TensorFlow AI
Hardware Required
# BeagleBone AI # Power Cable # External Fan (many blogs say that even though the BBAI runs hot, this is necessary: one example A simple start # Serial FTDI Cable # External Microphone (such as BeagleMic) # Camera (I have a Logitech USB webcam I could use, but for showing off what BBAI can do with images I'd probably choose the HD Camera Cape) # (if time or the scope of the project allows: BeagleBone-X-15)Timeline
Provide a development timeline with a milestone each of the 11 weeks and any pre-work. (A realistic timeline is critical to our selection process.)
Apr 8 | Found out about exciting projects at Beagle Board, joined the community! |
Apr 13 | Proposal complete, Submitted to https://summerofcode.withgoogle.com |
May 17 | Proposal accepted or rejected |
All of May | Prework: * Learn more about Linux Kernels * Try out TensorFlow Lite on Raspberry Pi that I already have so that I know how it is supposed to work * Brainstorm ideas for cool examples * Learn more about how TensorFlow Lite works on microcontrollers through the TensorFlow Lite google-group (that I've already joined) and other sources |
Jun 07 | Pre-work complete, Coding officially begins! |
Jun 17 | Milestone #1, Introductory YouTube video |
June 24 | Milestone #2, Working version of TensorFlow Lite on a BeagleBone AI |
June 30 | Milestone #3, Smooth(er) process and documentation for others to use TensorFlow Lite on BeagleBone AI |
July 12 18:00 UTC | Milestone #4, Is Milestone #3 as user friendly as possible? If not, improve it |
July 23 | Milestone #5, Example Cases on BeagleBone AI (vision) |
July 30 | Milestone #6, Example Cases on BeagleBone AI (sound) |
Aug 06 | Milestone #7, explore whether TensorFlow Lite can also be integrated with the BeagleBoard-X-15, and/or whether what I've learned about TensorFlow Lite can be extended to PyTorch or mlpack |
August 10 | Milestone #8, Completion YouTube video |
August 16 - 26 18:00 UTC | Final week: Students submit their final work product and their final mentor evaluation |
August 23 - 30 18:00 UTC | Mentors submit final student evaluations |