BeagleBoard/GSoC/2020Proposal/PrashantDandriyal 2
The page contains the second proposal to the initial project proposal for YOLO Models on x15/AI. The first proposal can be found here
Contents
BeagleBoard/GSoC/Proposal : YOLO models on the X15/AI
{{#ev:youtube|Jl3sUq2WwcY||right|BeagleLogic}} The project aims at running YOLO (v2 tiny) model inference on the BeagleBone AI at an improved rate of ~30 FPS by leveraging the on-board hardware accelerators and inference optimisations through TIDL API and TIDL Library. The model is to be converted and imported to suit the API requirements.
Student: Prashant Dandriyal
Mentors: Hunyue Yau
Code: https://github.com/PrashantDandriyal/GSoC2020_YOLOModelsOnTheX15
Wiki: https://elinux.org/BeagleBoard/GSoC/2020Proposal/PrashantDandriyal
GSoC entry
Status
This project is currently just a proposal.
=Proposal=sted on the ideas page and have created a pull request #135.
About you
IRC: pradan
Github: PrashantDandriyal
School: Graphic Era University, Dehradun
Country: India
Primary language:
Completed the requirements liEnglish, Hindi
Typical work hours : 12PM-6PM IST
Previous GSoC participation: None. The aim of participation stays bringing inference to edge devices; bringing the computation to the data than the other way round.
About your project
Project name: YOLO models on the X15/AI
Description
In 10-20 sentences, what are you making, for whom, why and with what technologies (programming languages, etc.)? (We are looking for open source SOFTWARE submissions.)
The project objectives can be fulfilled by approaching two paths: either optimising the model (path 1) or the inference methods (path 2). We propose to follow both the path with limited scopes, as defined in the upcoming sections. Texas Instruments provides an API: TIDL (Texas Instruments Deep Learning) to follow both the paths. For path 1, we intend to use the API for converting the darknet-19 based model into the intermediate format accepted by TIDL. We target only the YOLO v2-tiny model over the YOLO v3-tiny models as currently, not all layers are supported by the API. Also, the v2 model fits into the on-chip memory of the BeagleBone AI and the x15. The v2 is available in MobileNet and Caffe versions, both of which are supported for model import by TIDL. The Python method for a similar model import is implemented as follows:
python "tensorflow\python\tools\optimize_for_inference.py" --input=mobilenet_v1_1.0_224_frozen.pb --output=mobilenet_v1_1.0_224_final.pb --input_names=input --output_names="MobilenetV1/Predictions/Softmax"
The model import will certainly produce satisfactory results, using the techniques employed in _path 2_(discussed later). For a further improvement in performance, we plan to re-train the model using the /caffe-jacinto caffe fork. The validity of the performance is shown in the [1] section. This Texas Instruments-ported framework offers the following benefits:
- Efficient CNN configuration (grouped-convolution method)
- Introducing Sparsity and Quantization in models
- Layer grouping (for instance, the Convolution and ReLu layer are clubbed for reducing processing cost)
For Path 2, we use the TIDL library (a part of TIDL API) to modify how the inference is made. The /BeagleBone AI offers at least 2 Embedded Vision Engines (EVEs) and 2 C66x DSPs which help accelerate the frame processing by using multicore operation (with ~64MB memory buffer per core). We propose to use it for splitting the network layers between EVE subsystem and/or C66x cores. For example, to utilise all 4 accelerators (2 EVEs + 2 DSPs) for four different frames concurrently, arguments are passed:
./tidl_classification -g 2 -d 1 -e 2 -l ./imagenet.txt -s ./classlist.txt -i 1 -c ./stream_config_j11_v2.txt
where arguments -g and -e denote the number of EVEs and DSPs respectively. The software stack is shown in the below figure
Timeline
Provide a development timeline with a milestone each of the 11 weeks and any pre-work.
| April 27 | Pre-Work | Community Bonding Period and discussion on the project and resources available. |
| May 25 | Milestone #1, | - Introductory YouTube video
- Discuss and decide the scope of the project with the mentors - List and try collecting all the resources (documentation, hardware, etc) |
| June 1 | Milestone #2 | - Setup environment for development |
| June 8 | Milestone #3 | - Convert and maintain intermediate format models to be used
- Test the model by running inference locally (possibly emulating using executables) |
| June 19 | milestone #4 | - Demonstrate improved performance by running local system inferences and comparing with previous works.
- Document it - Submit report for Phase 1 Evaluation |
| June 22 | Milestone #5 | - Discuss modifications for the project plan with the mentors |
| June 29 | Milestone #6 | - Finalise the model to be used after performance results comparison
- Run first test on BeagleBone board - Document the issues/modifications made |
| July 6 | Milestone #7 | - Optimise the image feeding method
- Optimise the other parts of the pipeline |
| July 13 | Milestone #8 | - Test on image and video data
- Gather performance results and compare with previous works - Plan scope of second evaluation report |
| July 17-20 | Milestone #9 | - Submit second evaluation report
- Look for opportunity to use on-board hardware accelerators to further improve performance - Discuss possible improvements with mentors |
| July 27 | Milestone #10 | - Look for potential combination of hardware accelerators and TIDL and document it |
| August 3 | Milestone #11 | - Completion YouTube video
- Detailed project tutorial |
| August 10 - 17 | Final week | - Get the final report reviewed by mentors and improvise changes advised
- Submit final report |
Experience and approach
In 5-15 sentences, convince us you will be able to successfully complete your project in the timeline you have described.
Contingency
What will you do if you get stuck on your project and your mentor isn’t around?
Benefit
If successfully completed, what will its impact be on the BeagleBoard.org community? Include quotes from BeagleBoard.org community members who can be found on http://beagleboard.org/discuss and http://bbb.io/gsocchat.
Misc
Please complete the requirements listed on the ideas page. Provide link to pull request.
Suggestions
Is there anything else we should have asked you?
- ↑ Approach
