Audio Video Graphics Spec 5fR2

Draft 0.3

Introduction
Audio, video, and graphics processing is at the core of many CE products. The AVG requirements for CE devices are different than those for PCs/Servers, notably with respect to footprint, input devices, interlacing, streaming, etc.. Multiple graphics planes and video planes may be combined using, e.g., alpha blending and animation.

Rationale
No single default/standard interfaces exist for AVG. Having a well defined, well supported interface for AVG devices will reduce fragmentation of solutions and encourage the CE community to develop solutions that apply to conforming interfaces, so that they can be deployed across a wider range of systems.

Compliance classifiers
Terminology conventions are adopted here as they are defined in IETF RFC 2119, "Key words for use in RFCs to Indicate Requirement Levels" (by S. Bradner, March 1997). A compliance classifier from the following set may be used:


 * [M]ust, Required, Shall: This is the minimum set of requirements. The CELF based products are expected to comply with these requirements when expressed in unconditional form. A conditional requirement expressed in the form, "If X, then Y must be implemented", means that the requirement "Y" must be met when the conditional aspect "X" applies to a given implementation.


 * [S]hould, Recommended: Recommended items are optional items that are strongly recommended for inclusion in CELF based products. The difference between "recommended" items and "optional" items, below, is one of priority. When considering features for inclusion in a product, recommended items should be included first.


 * [O]ptional, May: Optional items are suggestions for features that will enhance the user experience or are offered as a less preferred choice relative to another recommended feature. If optional features are included, they should comply with the requirement to ensure interoperability with other implementations.


 * E[X]pressly Forbidden: This term means that an item must not be incorporated in a CELF based product.

Platforms
[O] Three target platforms are used or under consideration:
 * Renesas SH4 host with SM501 graphics (See SzwgPlatform3)
 * TI OMAP (See SzwgPlatform1)
 * X86 generic with Matrox G450/550

For the first two, the System Size Spec R2 page has a full description under "Definition - Platform".

Audio Specification
[O] No additional Audio specifications have been defined. ALSA, defined in kernel 2.6, may be used. Further evaluation is required before it can be considered for recommendation (see work in progress). Future extensions relate to AV streaming and synchronization.

Video-in/Capture Specification
[O] No additional Video input (capture) specifications have been defined. !V4L2, as defined in kernel 2.6, may be used.

[O] Proprietary solutions may also be used for video capture and digital tuners if !V4L2 does not suffice.

[O] DirectFB may be used as a higher level API.

Note: Video output can be seen as an (interlaced) sub-set of graphics. See graphics specification below for more details.

Video-out/Graphics Specification
[S] The standard Framebuffer is recommended for use in embedded CE devices.

[O] DirectFB may also be used in combination with the framebuffer.

Extensions to both are under consideration (see work in progress).

Graphics formats
[O] The framebuffer supports CLUT, RGB and RGBA packet data formats, but not YCbCr[A]. Hardware capable of accelerating the display YCbCr[A] packed data may develop their own extensions to the framebuffer for now.

[O] Also, the DirectFB framework which supports these formats may be used.

Multi-plane support
[O] Graphics hardware capable of multiple planes may be implemented with a single or multiple device drivers with one device per plane e.g. /dev/fb0, /dev/fb1,.../dev/fb5 for a 5 plane capable device. Front-end based scalers are recommended to use the DirectFB framework.

[O] Back-end scalers may add ioctl's to their framebuffer drivers.

Work in progress
Both DirectFB and the Framebuffer can be extended with YCbCr formats and multi-plane blending features commonly found in embedded CE devices. However, it is likely that only one of them will be supported in the future.

Resolution Support
The recommended formats are:
 * 4:4:4 Equal number of samples of Y, Cb and Cr.
 * 4:2:2 Cb/Cr are subsampled by a factor of two horizontally.
 * 4:2:0 Cb/Cr are subsampled by a factor of two in both directions.
 * 4:1:1 Cb/Cr are subsampled by a factor of four horizontally (used in DV).

If any of these formats are used, the CCIR 601 standard must be used. It defines how the data is interleaved and the relative positions of the Cb/Cr samples in relation to the Y samples.

Memory representation
YCbCr may be stored in e.g a framebuffer in various ways:


 * packed YCbCrA 4:4:4 : 32-bit unit containing one pixel with alpha
 * packed YCbCr 4:2:2 : 16-bit unit, two successive units contain two horizontally adjacent pixels, no alpha
 * planar YCbCr[A] 4:2:2 : three [four] arrays, one for each component
 * semi-planar YCbCr 4:2:2 : two arrays, one with all Ys, one with Cb and Cr.
 * planar YCbCr[A] 4:2:0 : three [four] arrays, one for each component
 * semi-planar YCbCr 4:2:0 : two arrays, one with all Ys, one with U and Vs

Following CCIR601, only the packed formats are recommended, with the possible exception of a separate alpha plane in some cases (see ARIB [O6] proposal).

Font rendering

 * freetype [O5]

Basic 2D acceleration

 * lines (horz./vert. vs. anti-aliased lines)
 * rectangles (fill and copy)
 * pixmaps (bitblt, scaling)

Video format control

 * resolution
 * interlaced/progressive

Multi-plane support

 * Each plane is represented by... [/dev/fb0, /dev/fb1,...]
 * Additional API (ioctl) calls... [display order, placement, scaling,...]

DirectFB specification
DirectFB overview [G2] provides a list of currently supported features, summarized below.

Important Terminology

 * Surface:Memory region physically reserved for rendering pixels. Surfaces are used for regular rendering of pixels, sprites and so on.
 * Sub-surface:Sub-region of surface. No physical memory allocated.
 * Primary Surface:Visible screen in full screen mode.
 * Layer:Each layer is different video memory. They are alpha-blended and displayed.
 * Window/Windowstack:Each layer may have multiple window. Windowstack is a stack of windows. Each window has surface. Their locations and orders may be changed.

Resolution Support
Supported formats are:
 * 4:2:2 Cb/Cr are subsampled by a factor of two horizontally.
 * 4:2:0 Cb/Cr are subsampled by a factor of two in both directions.

Memory representation

 * packed YCbCr 4:2:2 : 16-bit unit, two successive units contain two horizontally adjacent pixels, no alpha
 * planar YCbCr 4:2:0 : three arrays, one for each component

Font rendering

 * DirectFB bitmap font
 * !TrueType (using !FreeType2)
 * No bold or italics support other than by specifying a different typeface from the same font family. For example, 'Times New Roman Regular' and 'Times New Roman Italic' correspond to two different faces.

Basic 2D acceleration

 * lines (anti-aliased)
 * rectangles (fill and copy)
 * triangle (fill and copy)
 * pixmaps (bitblt, scaling)
 * Per-pixel alpha blending (a.k.a. texture alpha)
 * Per-plane alpha blending (a.k.a. alpha modulation)
 * Colorizing (a.k.a. color modulation)
 * Source and destination color keying

Video format control

 * resolution
 * interlaced/progressive support

Multi-plane support

 * DirectFB layers (not surfaces) support the concept of planes.
 * Layer API is provided through IDirectFBDisplayLayer Interface.
 * Opacity is available through IDirectFBDisplayLayer::!SetOpacity.
 * IDirectFBDisplayLayer::!SetScreenLocation controls scaling of the plane. Back-End Scaler(BES) is used, for instance for Matrox. It requires hardware support.
 * Explicit Front-End Scaler(FES) is not available. Thus, stretched blit to the primary surface should be used.
 * To execute a specific graphics operation (e.g. blitting of a surface), the DirectFB driver will access the memory mapped io ports of the graphics hardware to submit the command to the acceleration engine (actual hardware acceleration is done entirely from user space).

GFX Card Driver

 * DirectFB abstracts the video driver through GFX driver.
 * Graphic operation is executed through IDirectFBSurface interface. The interface calls appropriate callback routine in gfxcard driver(src/core/gfxcard.c). The callback routine decides whether the video device has hardware acceleration capability or not, and invokes appropriate functions.
 * The following is the model used in the core gfxcard driver. Blit, !DrawLine,

DrawRect and similar operations are implemented in this way:

void dfb_gfxcard_OPERATION {	bool hw == false; lock; /* check if acceleration is available, and then acquire */ if (hardware_accel_available(OPERATION) && hardware_accel_acquire(OPERATION)) { hw == card->funcs.OPERATION; }	/* if hardware acceleration is not available */ if (!hw) { gAcquire; gOPERATION; gRelease; }	unlock; }

DirectFb benchmarks
You can refer 'DirectFB' benchmark on various environment from Benchmark section of Evaluate Direct Fb Task Page

Remaining Issues
See Work in progress.