Thread:User talk:Jduchniewicz/GSoC 2021 Proposal: YOLO models on the X15/AI (2)

=YOLO models on the X15/AI = About Student: Jakub Duchniewicz Mentors: Hunyue Yau Code: not yet created! Wiki: https://elinux.org/BeagleBoard/GSoC/ProposalTemplate <- for now in a personal discussion board GSoC: YOLO Models on the X15/AI

=Status= Discussing the tentative ideas with Hunyue Yau on IRC.

=Proposal= Please complete the requirements listed on the ideas page and fill out this template.

About you
IRC: jduchniewicz Github: JDuchniewicz School: University of Turku/KTH Royal Institute of Technology Country: Finland/Sweden/Poland Primary language: Polish Typical work hours: 8AM-5PM CET Previous GSoC participation: Participating in GSoC, especially with BeagleBoard would further develop my software and hardware(BB X15/AI architecture) skills and help me apply my current knowledge for the mutual benefit of the open source community. I aim to deliver a component which will be usable in many upcoming releases of YOLO model and hopefully other models.

About your project
Project name: YOLO models on the X15/AI (with an extensible interface for other models)

Description
The main idea of the project is to accelerate Deep Learning models with help of available hardware resources on the BB X15 and BB AI platforms. Current inference times are abysmal for any real-time (or even slightly laggy but bearable) application ranging from 15 to 35 seconds per frame. This is unacceptable and this project will alleviate this problem and enable efficient deployment of other models once the Texas Instruments Deep Learning library allows for that (RNNs, LSTMs and GRUs are planned to be released).

As more and more developers recognize the benefits coming from DL and utilizing specialized hardware for acceleration of these calculations, the inclusion of such support is vital for BeagleBoard community. Additionally, adding such a component may encourage new developers interested in developing DL solutions for embedded systems to join the effort and grow the BB developer community.

/Begin unstructured CURRENT DOUBTS: can all layers be accelerated with TIDL? How to integrate it with the rest of ecosystem? - kernel interaction needed?? userspace wrapper? Maybe accelerate some steps with ARM NEON? if not possible?? Is building an extensible interface in scope for the solution? Maybe try some additional acceleration for non-AI platforms Probably a library of various supported networks? -> wrapping the execution model somehow so user can choose: I want YOLO model on this particular HW esssa What about utilizing DSPs Double buffering also? First answer doubts about how to structure this /End unstructured

The main scope of the project is to deploy YOLOv3 (tiny?) model and utilize the TIDL library C++ bindings to make it run smooth and efficient.

In 10-20 sentences, what are you making, for whom, why and with what technologies (programming languages, etc.)? (We are looking for open source SOFTWARE submissions.)

Timeline
Timeline needs more info of detailed steps, still not enough data. For sure setting up the enironment, deploying sample programs on the board, creating a scaffolding and only then creating the YOLO model bindings OR doing a very quick deployment of the YOLO and then creating proper abstractions. Provide a development timeline with a milestone each of the 11 weeks and any pre-work. (A realistic timeline is critical to our selection process.)

Experience and approach
I have strong programming background in the area of embedded Linux/operating systems as a Junior Software Engineer in Samsung Electronics during December 2017-March 2020. Additionally I have developed a game engine (PolyEngine) in C++ during this time and gave some talks on modern C++ during my time as a Vice-President of Game Development Student Group "Polygon".

Apart from that, I have completed my Bachelors degree at Warsaw University of Technology successfully defending my thesis titled: FPGA Based Hardware Accelerator for Musical Synthesis for Linux System. In this system I created a polyphonic musical synthesizer capable of producing various waveforms in Verilog code and deployed it on a De0 Nano SoC FPGA. Additionally I wrote two kernel drivers - one encompassed ALSA sound device and was responsible for proper synchronization of DMA transfers.

In my professional work, many times I had to complete various tasks under time pressure and choose the proper task scoping. Basing on this experience I believe that this task is deliverable in the mentioned time-frame.

In 5-15 sentences, convince us you will be able to successfully complete your project in the timeline you have described.

Contingency
Since I am used to tackling seemingly insurmountable challenges, I will first of all keep calm and try to come up with alternative approach if I get stuck along the way. The internet is a vast ocean of knowledge and time and again I received help from benevolent strangers from reddit or other forums. Since I believe that humans are species, which solve problems in the best way collaboratively, I will contact #beagle, #beagle-gsoc and relevant subreddits (I received tremendous help on /r/FPGA, /r/embedded and /r/askelectronics in the past).

If all fails I may be able be forced to change my approach and backtrack, but this will not be a big problem, because the knowledge won't be lost and it will only make my future approaches better. Alternatively, I can focus on documenting my progress in a form of blogposts and videos while waiting for my mentor to come back to cyberspace.

Benefit
The BB X15 and BB AI will be able to perform inference using YOLOv3 models in near real-time (maybe even allowing for using these boards for complex computer vision tasks). Additionally the BeagleBoard software codebase will have a good interface for deploying other models which would abstract the lower details of TIDL(or some other library future boards may use) interactions. The software will be prepared for rollout of newer and more advanced models.

Quotes?? If successfully completed, what will its impact be on the BeagleBoard.org community? Include quotes from BeagleBoard.org community members who can be found on http://beagleboard.org/discuss and http://bbb.io/gsocchat.

Misc
The PR is available here.

Suggestions
Is there anything else we should have asked you?