Difference between revisions of "BeagleBoard/GSoC/2021 Proposal/TensorFlow Lite Compatibility with BeagleBone AI"
Lpillsbury (talk | contribs) (→About you) |
Lpillsbury (talk | contribs) (→Benefit) |
||
Line 82: | Line 82: | ||
===Benefit=== | ===Benefit=== | ||
− | BeagleBoard options are strong contenders with Raspberry Pi and Arduino as tools that are easy to use to do IoT and industrial projects. While BeagleBones are geared more towards engineers and have some special features that actually make them quite different than the others (PRUs, ability to do low level and high level control simultaneously, power usage etc), they often get compared with the other frameworks. Given that Raspberry Pi and Arduino have TensorFlow Lite compatibility, I think it is important for BeagleBones to do the same. Also, there are algorithms that are easier to access with TensorFlow Lite than in other settings. | + | BeagleBoard options are strong contenders with Raspberry Pi and Arduino as tools that are easy to use to do IoT and industrial projects. While BeagleBones are geared more towards engineers and have some special features that actually make them quite different than the others (PRUs, ability to do low level and high level control simultaneously, power usage etc), they often get compared with the other frameworks. Given that Raspberry Pi and Arduino have TensorFlow Lite compatibility, I think it is important for BeagleBones to do the same. Also, there are some algorithms that are easier to access with TensorFlow Lite than in other settings. |
For example, | For example, | ||
Line 89: | Line 89: | ||
:-Jakub Duchniewicz | :-Jakub Duchniewicz | ||
− | + | In large laboratories and plants, experts often rely on the “sound” of machines to know quickly that a system is working properly. And most often small changes in pulsing tones or anomalies in the sound environment keys us into identifying problems. Much of this can be automated with machine learning with environmental monitoring sensors. I'd like to train a system to recognize when my helium reliquifiers or pulse tube coolers aren't functioning properly. | |
+ | :-Ritoban Basu Thakur, Caltech physicist [https://www.slac.stanford.edu/exp/cdms/ScienceResults/Theses/basuthakur.pdf Section 9.2 on audio monitoring for cryogenic equipment] | ||
and several frustrated users have posted on the internet: | and several frustrated users have posted on the internet: | ||
Line 107: | Line 108: | ||
The error is because TIDL does not support importing tensorflow lite model. Please try with Caffe/tensoflow/onnx models, for more details refer to TIDL datasheet/user-guide. | The error is because TIDL does not support importing tensorflow lite model. Please try with Caffe/tensoflow/onnx models, for more details refer to TIDL datasheet/user-guide. | ||
:-Praveen | :-Praveen | ||
+ | |||
+ | |||
+ | In addition to the requested need for TensorFlow Lite, this is something that BeagleBoard has already promised: | ||
+ | We've got a path to deliver the best AI training platform available, but, the actual materials are deeply lacking today. The https://github.com/beagleboard/cloud9-examples repository has a starting point for using the TIDL library, but that library is based on C/C++ and requires separate information on training and converting the model from Tensorflow/Caffe/etc. The TI Tensorflow Lite support is coming around the first quarter of 2020. | ||
+ | |||
+ | So, this is a work in progress. We'll be integrating a ton of examples (and associated software tools) to work with stuff like Tensorflow and Coral Accelerator in the short term. The expectation is to both crowd source this around the cloud9-examples repository and grow this over time. The hope is the Cloud9 development environment and mjpg-streamer presentation layer is enough inspiration and platform to get this moving quickly. | ||
+ | |||
+ | Today, conversion from Tensorflow or Caffe models to TIDL such that you can run them accelerated is covered at http://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/Foundational_Components_TIDL.html. You can also choose to run Tensorflow or Caffe natively on ARM. | ||
+ | :-[https://github.com/beagleboard/beaglebone-ai/wiki/Frequently-Asked-Questions#where-are-the-ai-training-materials] | ||
==Misc== | ==Misc== |
Revision as of 22:00, 11 April 2021
Contents
TensorFlow Lite Compatibility with BeagleBone AI
About
Student: Leah Pillsbury
Mentors: Contacted Robert C Nelson
Code: TensorFlow Lite Support
Wiki: Proposal/TensorFlow Lite Compatibility with BeagleBone AI
GSoC: N/A
Status
This project is currently just a proposal.
About you
IRC: lpillsbury
Github: lpillsbury
School: Pasadena City College (Boston University prior to PCC)
Country: United States
Primary language : English
Other languages : Spanish, Swahili, Hebrew, some Hindi/Urdu, Telugu, Bengali
Typical work hours : 9AM-5:30PM US Pacific, though I may be on Eastern time part of the summer
Previous GSoC participation: First time participation in GSOC. I am interested in learning more about embedded software, and participating in the community. I use open source tools regularly, and it is exciting to make something useful that other people would use too.
About your project
Project name: TensorFlow Lite Compatability with BeagleBone AI
Description
While the specs of BeagleBone AI are awesome, it seems limiting for an AI board for the general public to not be able to access TensorFlow Lite. I would like to change that by choosing a current release of TensorFlow, and writing the code necessary to run it on the Arm M4 processors that the BeagleBone AI board has, and that should work with TensorFlow Lite. This project will be a combination of C coding, dealing with the Linux kernel, and doing some example cases with TensorFlow Lite in python. Obviously, given the diversion between OpenCL and TensorFlow, there is no perfect way to seamlessly integrate TensorFlow Lite, but there seem to be enough people in the community who want it, that it is worth at least making a gesture towards a compatibility solution. I first realized this was a problem when I decided I wanted to learn more about AI on the edge. One of the first things that comes up when I search for information on this topic is using TensorFlow Lite and the wide variety of other boards that it plays well with.
Ideally, I'd spend the first few weeks getting TensorFlow Lite working on a BBAI, document my hacking process, and then create a smooth stable way to make it work more out of the box. Then I'd spend the rest of the time doing examples that would be highlighted both on the BeagleBoard website, and also on the TensorFlow Lite Examples page by making a pull request to the owners of the github repo. Given that there are already examples of using TensorFlow Lite on Raspberry Pi, with picamera, the starting point would be to have equivalent BeagleBone examples, most likely doing image capture through OpenCV (OpenCV with BeagleBone Black). I'm also very interested in sound recognition, and if time allows I'd also like to do some examples in this area.
Timeline
Provide a development timeline with a milestone each of the 11 weeks and any pre-work. (A realistic timeline is critical to our selection process.)
Apr 8 | Found out about exciting projects at Beagle Board, joined the community! |
Apr 13 | Proposal complete, Submitted to https://summerofcode.withgoogle.com |
May 17 | Proposal accepted or rejected |
All of May | Prework: * Learn more about Linux Kernels
|
Jun 07 | Pre-work complete, Coding officially begins! |
Jun 17 | Milestone #1, Introductory YouTube video |
June 24 | Milestone #2, Working version of TensorFlow Lite on a BeagleBone AI |
June 30 | Milestone #3, Smooth(er) process and documentation for others to use TensorFlow Lite on BeagleBone AI |
July 12 18:00 UTC | Milestone #4, Is Milestone #3 as user friendly as possible? If not, improve it |
July 23 | Milestone #5, Example Cases on BeagleBone AI (vision) |
July 30 | Milestone #6, Example Cases on BeagleBone AI (sound) |
Aug 06 | Milestone #7, explore whether TensorFlow Lite can also be integrated with the BeagleBoard-X-15, and/or whether
what I've learned about TensorFlow Lite can be extended to PyTorch or mlpack |
August 10 | Milestone #8, Completion YouTube video |
August 16 - 26 18:00 UTC | Final week: Students submit their final work product and their final mentor evaluation |
August 23 - 30 18:00 UTC | Mentors submit final student evaluations |
Experience and approach
In 5-15 sentences, convince us you will be able to successfully complete your project in the timeline you have described.
Contingency
What will you do if you get stuck on your project and your mentor isn’t around?
Benefit
BeagleBoard options are strong contenders with Raspberry Pi and Arduino as tools that are easy to use to do IoT and industrial projects. While BeagleBones are geared more towards engineers and have some special features that actually make them quite different than the others (PRUs, ability to do low level and high level control simultaneously, power usage etc), they often get compared with the other frameworks. Given that Raspberry Pi and Arduino have TensorFlow Lite compatibility, I think it is important for BeagleBones to do the same. Also, there are some algorithms that are easier to access with TensorFlow Lite than in other settings.
For example,
I tried tackling YOLO deployment on BBAI but it is currently impossible because of TIDL library restrictions. TFLite is the way to go with DL on BBAI and it uses TIDL underneath
- -Jakub Duchniewicz
In large laboratories and plants, experts often rely on the “sound” of machines to know quickly that a system is working properly. And most often small changes in pulsing tones or anomalies in the sound environment keys us into identifying problems. Much of this can be automated with machine learning with environmental monitoring sensors. I'd like to train a system to recognize when my helium reliquifiers or pulse tube coolers aren't functioning properly.
- -Ritoban Basu Thakur, Caltech physicist Section 9.2 on audio monitoring for cryogenic equipment
and several frustrated users have posted on the internet:
I've been attempting to build the TF Lite library on my BeagleBone AI (32-bit ARMv7 MCU) for several days now, to no avail.
- -Matt
I wish to run my network on the AM5729 found on the BeagleBone AI board. I understand that the ti-processor-sdk-linux-am57xx-06.03.00.106 does not yet support the BeagleBone AI tool. However, the Debian image distributed for the BeagleBone AI has all the TIDL libraries packaged with it.
Therefore the only thing I need to be able to do is convert the tensorflow lite network into the TIDL format using the TIDL import tool. I assume I can do this using the following flow: 1. download the ti-processor-sdk-linux-am57xx-06.03.00.106-linux-x86-Install.bin from TI 2. run the installer on a linux (Ubuntu) host 3. navigate to the tidl_model_import.out tool 4. run the tidl_model_import.out tool on an appropriate configuration file (below)
- -Alex Beasley
The error is because TIDL does not support importing tensorflow lite model. Please try with Caffe/tensoflow/onnx models, for more details refer to TIDL datasheet/user-guide.
- -Praveen
In addition to the requested need for TensorFlow Lite, this is something that BeagleBoard has already promised:
We've got a path to deliver the best AI training platform available, but, the actual materials are deeply lacking today. The https://github.com/beagleboard/cloud9-examples repository has a starting point for using the TIDL library, but that library is based on C/C++ and requires separate information on training and converting the model from Tensorflow/Caffe/etc. The TI Tensorflow Lite support is coming around the first quarter of 2020.
So, this is a work in progress. We'll be integrating a ton of examples (and associated software tools) to work with stuff like Tensorflow and Coral Accelerator in the short term. The expectation is to both crowd source this around the cloud9-examples repository and grow this over time. The hope is the Cloud9 development environment and mjpg-streamer presentation layer is enough inspiration and platform to get this moving quickly.
Today, conversion from Tensorflow or Caffe models to TIDL such that you can run them accelerated is covered at http://software-dl.ti.com/processor-sdk-linux/esd/docs/latest/linux/Foundational_Components_TIDL.html. You can also choose to run Tensorflow or Caffe natively on ARM.
- -[1]
Misc
Please complete the requirements listed on the ideas page. Provide link to pull request.
Suggestions
Is there anything else we should have asked you?