BeagleBoard/GSoC/2022 Proposal/Running Machine Learning Models on Bela

=Running Machine Learning Models on Bela= About Student: Ezra Pierce Mentors: Jack Armitage, Victor Shepardson Proposal:

=Proposal=

=Status= This project is currently just a proposal.

About you
Github: ezrapierce000 School: [Carleton University] Country: Canada Primary language : English Typical work hours: 9AM-6PM Eastern Standard Time Previous GSoC participation: This would be my first time participating in GSoC.

About your project
Project name: Running Machine Learning Models on Bela

Introduction
The usage of machine learning in musical instrument design has grown in recent years, yet there has not been many implementations in more resource-constrained embedded contexts. The goal of this project is to improve the tooling surrounding embedded machine learning on the Bela to aid its community in experimenting with machine learning applications in their audio projects. The specific developer tool chosen for this project is an inference benchmarking tool developed for the BBB/Bela platform. Alongside this tool, the project will also build up a model zoo for the Bela and build some example projects to give solid starting point to those just starting out.

Model Selection
During the community bonding period, some models will be selected to cover potential use cases such as:
 * Gesture recognition
 * Control mapping
 * Sound synthesis

These are unique use cases that are not suited to the usual models found in other ML benchmarking tools which tend to be focused on computer vision. Some models for these use cases may also be hindered by the limited operator support in common embedded ML frontends, this will be taken into consideration during the model selection process to pinpoint which frontends may be best suited for different use cases. The models selected for the model zoo will also form the basis for example projects developed later in the project.

Proposed model architectures:


 * 1D convolutional network (for audio or sensor streams)
 * memory-cell RNN (e.g. GRU) (for audio or sensor streams)
 * MLP (general purpose)
 * Transformer block (general purpose)
 * mixture-density head (with many elementwise, shape and reduction ops; +with RNG in graph

Implementations
Focus is on a specific modeling langauge (pytorch) and platform (BBB+Bela). In between there are a number of model formats, compilers/runtimes frontents, and backend components to investigate.

summary of stack:


 * Modeling language: pytorch. (+tensorflow for converting to tflite)
 * Model format: ONNX, torchscript, (+tflite)
 * Runtime frontends: libtorch, ONNX runtime, SOFIE, (+tflite)...
 * Runtime backend components: ArmNN, XNNPack, eigen, BLAS, ...
 * OS + Hardware: Bela + BBB.

more NN compiler projects to quickly audit for activity / potential BBB support:


 * torch-MLIR (https://github.com/llvm/torch-mlir)
 * plaid (https://plaidml.github.io/plaidml/)
 * glow (https://github.com/pytorch/glow)
 * NNC (https://dev-discuss.pytorch.org/t/nnc-walkthrough-how-pytorch-ops-get-fused/125)
 * Apache TVM (https://tvm.apache.org/)

Benchmarking
In parallel to the development of the model zoo, a benchmarking suite will be developed to evaluate ML models running on the BBB/Bela. This benchmarking suite will allow for a developer to load a model & test data from their host PC onto the Bela via a simple Python utility, then run performance and accuracy tests on the Bela.

The benchmarking suite will be written in C++ and take latency & memory usage measurements as the model is run.

TODO: flesh out implementation details

Contingency
if I get stuck on my project and my mentor isn’t around, I will use the following resources: