Multimedia

Introduction
Here are some miscellaneous resources related to audio, video and graphics systems under Linux:

Also see the section on User Interfaces.

CELF 2.0 Specification for AVG
(more like a set of recommendations rather than a specification)

Audio Video Working Group
Please see the CELF wiki for more information: Audio Video Graphics Working Group

Some AVWG related Outdated pages

What is DirectFB, How Does DirectFB Work
DirectFB

Sample Implementation of DirectFB on an embedded Linux platform
Porting DirectFB

Some DirectFB benchmark on embedded Linux platform
Benchmark DirectFB

Framebuffer

 * http://www.kernel.org/ (1) KD26/fb
 * http://linuxconsole.sourceforge.net/fbdev/HOWTO/
 * http://www.tldp.org/HOWTO/Framebuffer-HOWTO.html

Stores the frame information in the videos ....

DirectFB

 * http://www.directfb.org/
 * http://www.directfb.org/documentation/DirectFB_overview_V0.2.pdf
 * DirectFB

V4L2

 * http://www.linuxtv.org/
 * http://www.linuxtv.org/downloads/video4linux/API/V4L2_API/spec-single/v4l2.html

X11

 * http://www.x.org/
 * X11

NanoX

 * http://www.microwindows.org/

OpenGL (OpenML)

 * http://www.opengl.org/
 * http://www.khronos.org/opengles/

SDL

 * http://www.libsdl.org/ immediate renderer library with very bare bones primitives like rectangle fill and blit. Since it exposes just framebuffer and few primitives, it's easy to port to different platforms, actually it was born as a way to port Windows games to Linux.

Cairo

 * http://www.cairographics.org/ is an immediate renderer library that can do complex vector graphics, including matrix transforms. It runs on top of DirectFB, X11, memory buffers and more. It is the base of some toolkits like GTK and applications like Firefox.

Clutter

 * http://clutter-project.org/ is an object-oriented 3d canvas on top of OpenGL (or OpenGL-ES) with scene management. It is based on GLib/GObject and matches nicely GNOME platform. Many powerful Linux mobile devices will ship with Clutter-based intefaces in near future, like Intel's Moblin, Ubuntu Mobile and Maemo.

Enlightenment Foundation Libraries (EFL)
The Enlightenment Foundation Libraries contains Evas, an object-oriented 2d canvs on top of OpenGL/X11, XRender/X11, X11, FB, DirectFB, DirectDraw and more. It includes scene management and integrates with Ecore, matches nicely other EFL components like Edje. It's used by some media centers and the OpenMoko phone. See Gustavo Barbieri conference at Embedded Linux Conference Europe 2008, slides and video. Gustavo's company, ProFUSION, offers services around EFL.

Qt

 * Qt is a crossplatform graphics toolkit with support for framebuffer and X. Has advanced animation capabilities using Graphics View framework.

Storyboard Suite (Storyboard Designer/Engine from Crank Software)

 * http://www.cranksoftware.com/storyboard The Storyboard Suite from Crank Software provides a complete environment to design, develop and deploy embedded user interfaces across multiple rendering technologies (DirectFB, FBDev, SDL, OpenGL, OpenVG, ...), multiple operating systems (Linux, QNX, VxWorks, WinCE/Win32, ...) and multiple architectures (x86, ARM, PPC, SH, ...). The Storyboard approach is unique in that it has been developed to incorporate content directly from graphic designers and deploy data bundles that are specifically optimized for each OS/CPU/Rendering technology set.

GStreamer

 * http://www.gstreamer.net/ is a open-source multimedia framework allowing the creation of multimedia applications by assembling processing nodes (called elements) in a graph (called pipeline). The range of plugins available allow easy creation of playback applications, recorders, audio/video editors, streaming servers, visioconference system. The variety of plugins range from decoders, encoders, muxer, demuxers, network sources for a variety of protocols, hardware accelerated features (decoding, display, capture,..), video filters. Its low-level flexibility also makes it sometimes complex to use, but is assisted by several convenience plugins linke playbin, decodebin, camerabin making simple use-cases easy to use. It is built on top of GLib/GObject, making it easily portable to any new platform. It is being used in more and more in embedded devices due to the availability of quality LGPL plugins for various format support, support for lip-sync, support for network streaming, standard linux API, and easy wrapability of hardware devices like DSP-accelerated codecs.

Xine

 * http://xinehq.de/ is a playback media engine that handles most of the complexity for you. It's based on threads, so clock and synchronization are handled automatically. Note that this library is GPL licensed so your application must be GPL compatible to use it.

MPlayer

 * http://mplayerhq.hu/ it's not a library but an application, however it's controllable from other applications and it's used as media framework for some systems. It's GPL as well as xine, but since it's externally controlled you don't need to make your application GPL to use it.

Documentation

 * Presentation Choosing embedded graphical libraries held by Thomas Petazzoni at the ELCE 2008

Video in
= V4L2 ELC Presentations =

Shifting Media App Development Into High Gear.
Presenter:Helen Kolke, Collabora Ltd. Summary: These slides cover the Classic V4L2 API which includes vivid driver and Media API (extension) which includes Vimc Driver.


 * [| [PDF ]] ELC 2018.

2017 is the Year of the Linux Video Codec Drivers.
Presenter:Laurent Pinchart, Ideas on Board Summary: This presentation explains the reasons behind delayed support for video codecs and the ways they can be implemented and used with free software and open APIs.


 * [| [PDF ]] [| [VIDEO ]] ELC 2017

V4L2 & CEC Status Report.
Presenter:Hans Verkuil, Cisco Systems Norway Summary: This presentation provides all the information about the new drivers in v4L2 Drivers and Upcoming V4L2 Drivers.


 * [| [PDF ]] [| [VIDEO ]] ELC 2017

V4L2 on Steroids: The Request API
Presenter:Laurent Pinchart Summary: This presentations explains the design of the Request API and how to use it in both applications and drivers. The focus will be put on two major use cases, 2D hardware composers and Android camera HAL v3 implementation.


 * [| [PDF ]] [| [VIDEO ]] ELC 2016

HDMI CEC: What? Why? How?
Presenter:Hans Verkuil, Cisco Systems Norway Summary: This Presentation includes a short introduction of the upcoming CEC framework and the utilities that use it.


 * [| [PDF ]] [| [VIDEO ]] ELC 2016.

Maximize the Performance of HTML5 Video in RPi2
Presenter:Gwang Yoon Hwang, Igalia Summary: This presentation talks about lessons learnt while executing rendering and composing pipeline of WebKitGTK+, WebKitForWayland, and GStreamerGL using OpenMAX and OpenGL in RPi2 device.


 * [| [VIDEO ]] ELC 2016

Testing Video4Linux Applications and Drivers
Presenter:Hans Verkuil Summary: This presentation covers the V4L2 compliance driver testing, Qt test application and Test Drivers Cropping


 * [| [PDF ]] ELC 2015.

Hardware Accelerated Video Streaming with V4L2.
Presenter:Hardware Accelerated Video Streaming with V4L2. Summary: This presentation gives introduction of Simple V4L2 applications and V4L2 application using OpenGL as well vendor specific features.


 * [| [PDF ]] ELC 2014.

An Introduction to the Video4Linux Framework.
Presenter:: Hans Verkuil, Cisco Systems Norway Summary: These slides cover the V4L2 Framework, V4L2 PCI Skeleton Driver Control Framework and Features of the Skeleton Driver.


 * [| [PDF ]] ELC 2014

Optimizing GStreamer Video Plugins: A Case Study with Renesas SoC Platform.
Presenter:Katsuya Matsubara, IGEL Co.Ltd Summary: These slides cover the case study with Remesas SoC Platform for Optimizing GStreamer Video Plugins and also contains the optimizing the video monitoring.


 * [| [PDF ]] ELC 2013.

Video4linux: Progress, New videobuf2 Framework and the Future.
Presenter:Hans Verkuil (Cisco) Summary: This presentation provides information about external video and audio codecs are linked to Stagefright through OpenMAX IL.


 * [| [PDF ]] ELC 2011

Bringing up HDMI Display for OMAP4 Panda Board - Design, Challenges and Lessons Learned.
Presenter:Mythri pk Summary: This Presentation Provide information about the high level software, audio data path, video data path,OMAP4 HDMI hardware and Use case of HDMI Hot-plug Enable.


 * [| [PDF ]] ELC 2011

Recent Developments in Open Video Technology.
Presenter:David Schleef Summary: This presentation provides the information of Recent Developments in open video technology and use of Open Video Technology.


 * [| [PDF ]] ELC 2010

A Linux multimedia framework for SH-Mobile processors.
Presenter:Conrad Parker Summary: These slides cover the working of VPU: Video Processing Unit and VOU: Video Output, Sound I/O I USB and related Linux Kernel interfaces required for inter-operation.


 * [| [PDF ]] ELC 2009

Video4Linux: What about Output?
Presenter:Matthew Porter Summary: These slides cover the video capture drives under a unified interface and usually considered as a video capture framework- Cameras, TV tuners and radio.


 * [| [PDF ]] ELC 2009

Basic video editing on embedded devices using GStreamer.
Presenter:Christian F.K. Schaller Summary: This presentation provides the overview of gstreaer, whats new since last year and multimedia framework.


 * [| [PDF ]] ELC 2009

Framework for digital camera in Linux.
Presenter:: Dongsoo Kim, HeungJun Kim Summary: This presentation provides information of the new V4L2 APIs for digital camera lens control, object recognition and abstract work flow of switching camera


 * [| [PDF ]] ELC 2009

Adding Framebuffer support for Freescale SoCs.
Presenter:York Sun Summary: This presentation introduces the Key Features of MPC8610 like Display Interface Unit (DIU), DIU Modes of Operations, DIU: Pixel Structure, Area Descriptors, Software Components used on MPC8610 HPCD.


 * [| [PDF ]] ELC 2008.

Video Clip Player – Philips Nexperia™ PNX0106 and Linux based platform.
Presenter:Armin Gerritsen Summary: This presentation covers the introduction of Video Clip Player, followed by System Overview and Use Cases. in addition to vide software architecture in Linux. It also covers Framebuffer Implementation ,VoIP application and its prototyping on Linux


 * [| [PDF ]] ELC 2006.

V4L[2]

 * http://www.kernel.org/ (1) KD26/video4linux
 * http://www.linuxtv.org
 * http://www.linuxtv.org/downloads/video4linux/API/V4L2_API/spec-single/v4l2.html

OpenML

 * http://www.khronos.org/openml/

LinuxTV (DVB API)

 * http://www.linuxtv.org

OSS

 * http://www.kernel.org/ (1) KD26/sound/oss
 * http://www.4front-tech.com/oss.html

ALSA

 * http://www.kernel.org/ (1) KD26/sound/alsa
 * http://www.alsa-project.org

OpenAL

 * http://www.openal.org/

PulseAudio

 * http://pulseaudio.org PulseAudio is a multi-platform sound server which brings a lot of cool features to Linux sound. Why you want to use it for for embedded systems (besides having support for cool stuff like bluetooth audio, apple airport etc) is described here: http://0pointer.de/blog/projects/pulse-glitch-free.html

Video Lan

 * http://www.videolan.org

Freevo

 * http://freevo.sourceforge.net

LinuxTV

 * http://www.linuxtv.org/

MythTV

 * http://www.mythtv.org/

DVR

 * http://dvr.sourceforge.net/html/main.html

OpenPVR

 * http://www.funktronics.ca/openpvr/
 * http://sourceforge.net/projects/openpvr/

Morphine.TV

 * http://wiki.morphine.tv
 * http://sourceforge.net/projects/mms4l/

ARIB architecture

 * http://www.arib.or.jp/english/html/overview/ov/std_b24.html

Boot Splash

 * www.bootsplash.org

Digital Home Working Group

 * http://www.dhwg.org/

Disko Framework

 * http://www.diskohq.org
 * http://www.directfb.org

Free Type

 * http://freetype.sourceforge.net/freetype2/

UPnP

 * see UPnP

TV Anytime

 * http://www.tv-anytime.org/

TV Linux Alliance

 * http://www.tvlinuxalliance.com/

Note (1) - KD26 refers to the Linux 2.6.X kernel tree, which has a "Documentation" sub-directory.