BeagleBoard/GSoC/2020 Projects/Media IP Streaming

=Proposol of equipping the Beaglebone AI with Media IP Streaming capabilities =

This project will equip the Beagleboard AI with Media IP Streaming capabilities, by porting the sound card drivers for CTAG face2|4 Audio Card and the AVB protocol stack from Beagle Bone AVB to the Beaglebone AI.

Student: nwan Mentors: rma Code: https://github.com/NiklasWan/linux Wiki: http://elinux.org/BeagleBoard/GSoC/MediaIpStreaming GSoC: [N/A]

=Status= This project is currently just a proposal.

=Proposal= All requirements have been fullfilled, the Pull Request can be found here #139

About you
IRC: nwan Github: NiklasWan School: Kiel University of Applied Sciences Country: Germany Primary languages: German, English Typical work hours: 8AM-5PM CET Previous GSoC participation: I want to participate at GSoC because I want to gather experience in working within an open source community and try to apply theoretical knowledge into the practical domain. Also I hope to learn new awesome things. This would be my first time participating in GSoC.

About your project
Project name: Media Ip Streaming

Description
The BeagleBone AI is equipped with a high amount of processing power due to the Dual Core ARM Cortex-A15 chip as a main computing unit and accompanying co-processors. This makes the AI a perfect fit for highly demanding applications regarding CPU consumption, like audio applications which have extremely strong realtime constraints. Professional audio/video studios have to guarantee for small latencies when transmitting media signals between different devices and different media channels in a transmitted stream need to be synchronized. Latency and Snychronicity is extremely important when transmitting e.g. a video channel together with the accompanying audio channel. Those two channels have to be transmitted in a manner, that lip synchronicity can be guaranteed because humans are extremely sensitive to voice offset to accompanying video signals.

To bring media ip streaming capabilities to the Beaglebone AI, the following steps are planned: A previous GSoC project ported a sound card driver from the BeagleBone Green/Black to the BeagleBoard-X15 (https://summerofcode.withgoogle.com/archive/2016/projects/5351212496977920/). This port will now be ported to the BegleBone AI. With the sound card driver successfully ported, the next step would be to port the AVB protocol driver stack from Beagle Bone AVB enabling media streaming over the network. This would allow to use the Beaglebone AI as a media streaming device in professional audio/media applications and bring audio stream synchronisation features to the Beaglebone AI. Thus allowing for tight synchronization between different audio and video streams which are transmitted over the network. Additionally for people who don't own the CTAG Face 2|4 cape HDMI audio output should be realized.

Practical Use: AVB is primarily used in large scale media productions, like sports venues, broadcasting studios or concert halls. Basically AVB can be used everywhere where media data has to be transmitted over larger distances in a local network. Implementing this on a Beaglebone AI would allow for a low cost alternative for proprietary hardware and further allow for customization by the Beagleboard.org community.

Timeline
Provide a development timeline with a milestone each of the 11 weeks and any pre-work. (A realistic timeline is critical to our selection process.)

Experience and approach
During my bachelor's degree in information technology I had several courses like programming in c, programming in c++, operating systems and embedded system programming which layed down the basis for developing embedded software. Due to my additional bachelor's degree in audio production I have additional experience in audio applications and audio and media codecs, which will help me to understand the theory behind the different needed algorithms. With my previous development work for the Strämpler project I already have experience in working on complex embedded c projects and which potential pitfalls could occur.

Contingency
If I get stuck and my mentor is not around I will follow the following steps in displayed order:
 * 1) Search the internet for the problem.
 * 2) Serach through literature acquired during milestone #1.
 * 3) Ask in the GSoC IRC, if fellow students know a solution to the specific problem.
 * 4) If the problem is still not solved, postpone the problem until mentor is available again and work on another part of the project.

Benefit
Equipping the Beaglebone AI with media ip streaming capabilities would allow the Beaglebone.org community to use those capabilities to implement the system in professional media applications. The community could also implement further media protocols like AES/Ravenna to allow the usage of the AI for even more media streaming tasks.

Misc
Link to pull request #139.

PTP Overview

 * is used by the AVB protocol to achieve synchronization between devices
 * based on IEEE 1588v2
 * up to nanoseconds accuracy
 * sharing timestamps over the network for synchronisation of devices
 * uses master/slave hierarchy
 * slave retrieves time from master ==> network dely has to be taken into account

Ordinary Clock

 * normally endpoint of the network
 * single port
 * BMCA determines which clock is used as master (the one with the highest accuracy)

Grandmaster Clock

 * is used as an endpoint master and has extremely high accuracy (normally timed by GPS or NTP)
 * there can be more than one in a network to achieve redundancy

Boundary Clock

 * mutli port
 * a network switch with master/slave ports

Transparent Clock

 * accounts for queing delays when a standrad switch is used and thus improves accuracy

E2E:

 * calculate network delay End-To-End
 * no need of PTP equipment but this results in added cost in accuracy

P2P:

 * calculate network delay Peer-To-Peer
 * results in high accuracy, but all devices in the network need to be PTP enabled