EBC Exercise 20 The Display SubSystem (DSS)

This is a four part lab that explore the video system drivers. The labs demonstrate the Linux V4L2 and FBdev drivers as well as basic file I/O, through four small applications: on-screen display (OSD), video recorder, video player, and video loop-thru (video-capture copied to video-display). It's based on the materials used in TI's DaVinci System Integration using Linux Workshop. The workshop is based on the DVEVM. I've converted those materials to the BeagleBoard.

This lab shows three ways to write directly to the framebuffer (/dev/fb0) on the Beagle:
 * 1) Write a single value to the entire framebuffer
 * 2) Write the pixel values based on a formula
 * 3) Read a bmp image and write it

The following labs will show how to bring in live video from a web cam via V4L2 and save it to a file, write video from a file to a video framebuffer (/dev/fb1), and finally move live video from a web cam to the framebuffer. A future lab may show how to process the video on the DSP before displaying it.

This is the first of 4 exercises:
 * Part a: You will build an on-screen display for the your device using the FBDEV driver – INSPECTION LAB only.
 * Part b: Examines v4L2 video capture via a simple video recorder application – INSPECTION LAB only.
 * Part c: Examines the v4L2 display driver using via a video display application. This application plays back the file captured in lab 07b – INSPECTION LAB only.
 * Part d: You will combine the recorder and player applications into a video loop-thru application using memcpy to transfer data between capture and display drivers.

Part a - OSD Setup
The goal of this part is to build an on_screen display for the Beagle using the FBDEV driver. From a coding perspective, it’s an inspection lab only.


 * 1) Create your own custom picture for the OSD window (using gimp), saving the picture to 32-bit format RGBA.
 * 2) Inspect video_thread.c and helper functions (inside video_osd.c).
 * 3) Build, run. Result: see your custom banner displayed on screen (no video yet…).

or if you've already git'ted them, make sure you have the most up to date versions. Either way This file is 640 (width) by 480 (height). All the code that follows assumes the image is this size. (You may change the size, but you'll also have to change the code.)
 * Get the files from the git repository
 * 1) git clone git@github.com:MarkAYoder/BeagleBoard-exercises.git exercises
 * 1) cd exercises
 * 2) git pull
 * Change to the directory: VideoThru/lab07a_osd_setup
 * Open the Gimp (open-source) paint program by typing “gimp Rose640x480a.bmp” in the terminal.
 * Edit the file to create a custom banner picture.


 * Paint something for your OSD banner. You can create a simple graphic quickly using just three of the many tools.
 * Before clicking any of the tools, you can choose a color first using the color box.
 * Start with the gradient tool to create a background. Select the tool, then click and drag the mouse over the 640x80 image area.
 * Add text or paint something over the gradient with either of these tools, respectively.
 * Save your file and exit Gimp.



When you are finished, save with File:Save. Then exit Gimp. The file should save as RGBA, that is red, green blue, alpha. Where alpha is the transparency. It DOES matter what you name the file because later, during the building process, this file is specifically copied to the target. This name also makes it easy to remember it’s a 640x80 Bitmap image in 24-bit RGBA.


 * Make sure your file is saved to your lab folder.


 * List the contents of the lab folder

video_osd.c
video_osd.c contains a number of functions for manipulating the on-screen display. These functions are not part of an official package, but were developed for these lab exercises to demonstrate the capabilities of the on-screen display hardware.
 * 1) video_osd_place: places a picture on the OSD display. Assumes data is provided in 32-bit ARGB (8-bit attribute transparency, 8-bit red, 8-bit blue 8-bit green per pixel).
 * 2) video_osd_scroll: a more complex version of video_osd_place that will offset the OSD display by x and/or y scroll values. This can be used to scroll a banner or picture horizontally or vertically.
 * 3) video_osd_circframe: draws a circular alpha-blended frame around the video output.

osd_thread.c
The thread function in osd_thread.c is video_thread_fxn which uses the helper functions from video_osd.c which supports alpha blending:
 * 1) calls video_osd_setup to open the OSD window. This window is memory mapped (mmap’ed) into the application space and a handle to the Display object is returned and stored in osdFd.
 * 2) calls fopen to read the custom banner picture (as created in gimp and stored in a 32-bit ARGB bitmap file).
 * 3) The OSD buffer is initialized by placing the picture buffer using video_osd_place, which places the picture on the OSD window with a y offset of 200).
 * 4) Also  a call is made to video_osd_circframe to initialize the OSD buffer with a circular semi-transparent (0x80 is 75% transparency) green frame (0x0000FF is blue, 0x00FF00 is green, 0xFF0000 is red).

The application assumes the picture is supplied in a 640 x 80 32-bit ARGB (8-8-8-8) format, which should be the case if you followed the previous gimp instructions.

$ make debug $ ./videoThru_DEBUG.Beagle What do you see?
 * Build and execute the application.

Part a Questions

 * 1) How would you modify the lab07a_osd_setup application to make the banner you created semi-transparent instead of solid?
 * 2) How would you modify the lab07a_osd_setup application to place your banner at the top of the screen?
 * 3) (Advanced) Why is the osdFb preceded with an ampersand (&) in the call to video_osd_setup( &osdFd, FBVID_OSD, 0x40, &osdDisplay )?

Part b - Video Capture
The goal of Part b is to examine v4L2 video capture via a simple video recorder app. This is an Inspection lab only.


 * 1) Examine helper functions (setup, cleanup, wait_for_frame) in video_input.c.
 * 2) Examine video_thread_fxn in video_thread.c.
 * 3) Examine main.c (how the signal handler is created/used, then calls video_thread_fxn).
 * 4) Build, run. Result: create a file (video.raw) that contains about 2s of captured video.


 * Change to the directory: lab07b_video_capture

video_thread.c
This file contains a single function, video_thread_fxn. This function encapsulates the functionality necessary to run the video recorder and is analogous to the audio_thread_fxn that was used in the previous lab.

video_thread_fxn utilizes the following:
 * video_input_setup
 * these function is used to detect the video standard which the web cam is using and calculate the corresponding buffer size of a single video frame. It opens and configures the Linux V4L2 video capture driver. The driver uses driver-allocated (as opposed to user-allocated) buffers that are mmapped to the user space to store video frames. Advanced users can take a look in video_input.c to see the details.


 * fopen
 * Standard Linux I/O (i.e from #include ) call to open a file where the captured video data will be written


 * for loop
 * Loops through 100 cycles so as not to overflow /tmp directory’s RAM memory
 * ioctl( captureFd, VIDIOC_DQBUF, &v4l2buf ): dequeues the next video frame from the V4L2 driver (blocks/pauses if buffer is not available, yet).
 * fwrite: The video frame is copied into the file.
 * ioctl( captureFd, VIDIOC_QBUF, &v4l2buf ): Once the application has finished writing the video buffer to the file, the buffer handle must be passed back to the driver so that it can be refilled with new video data.
 * ioctl( captureFd, VIDIOC_QBUF, &v4l2buf ): Once the application has finished writing the video buffer to the file, the buffer handle must be passed back to the driver so that it can be refilled with new video data.

main.c
This is the entry point for the application. main does the following:
 * Creates a signal handler to trap the Ctrl-C signal (also called SIGINT, the interrupt signal). When this signal is sent to the application, the videoEnv.quit global variable is set to true to signal the video thread to exit its main loop and begin cleanup.
 * After configuring this signal handler, main calls the video_thread_fxn function to enter into the video thread. Upon completion of this function, main checks the return value of the function (success or failure) and reports.

Build the application using make using “make debug”
Enter: $ ./videoThru_DEBUG.Beagle
 * Run the application

You will get a message from the application indicating that it has captured video frames. Check the following to ensure that the video has recorded properly: ls –lsa /tmp/video.raw The file should be about 60 MB in size. The reason that the application only records 100 video frames is to keep from overflowing the /tmp directory.

Part c - Video Playback
The goal of this part is to examine FBdev display driver using a video display app. This app will play back the file recorded in Part b (and add OSD from Part a). This is an inspection lab only.


 * 1) Examine video_output.c and its helper functions.
 * 2) Ensure /tmp/video.raw still exists.
 * 3) Build, run. Result: video.raw file is displayed on the screen (along with your OSD).


 * Change to lab07c_video_playback.

Examine video_thread.c
As opposed to the recorder, this application uses ioctl calls to manage the framebuffer at /dev/fb1 to display video frames it reads from the /tmp/video.raw file:
 * fopen and fread
 * Opens the input file containing captured video frames.


 * video_osd_setup: Creates a table of appropriately sized buffers to hold video frame data that is read from the file. The driver uses driver-allocated buffers which are mapped user space to store video frames.

video buffer.
 * while loop:
 * Loops until ctrl-C is pressed or input file is depleted
 * fread: The next video frame is read from the file and copied into the
 * flip_display_buffers: Uses ioctl calls to tell the display subsystem to display the next buffer.


 * Build and run the application.

Use “ls –lsa /tmp/video.raw” to verify that video.raw exists and has a greater than zero file size. The application is hard coded (using a #define statement in video_thread.c) to read data from the file /tmp/video.raw. If you have powered off or reset since running Part b, the video.raw file will have been cleared. If so, go back to lab07b_video_capture to create the video.raw file again.
 * Check to make sure /tmp/video.raw exists and has a file size larger than zero.

The Display Subsystem (DSS)

 * Execute ./videoThru_DEBUG.Beagle. The application should play back the video from /tmp/video.raw along with your customized OSD banner.

If you just run the application, you will see the bmp file from before, but you won't see any video. The OSD stuff is being written onto /dev/fb0, that is framebuffer 0, the same framebuffer Linux is using. The video is being written to /dev/fb1, one of the two video framebuffers. Currently that framebuffer isn't being displayed.

Do the following: $ cd .. $ ./vid1Show $ cd lab07c_video_playback $ ./videoThru_DEBUG.Beagle You now see the video playing.

gedit vid1Show and figure what it's doing. Change this line echo 500,100 > $ovl1/position to this echo 100,100 > $ovl1/position and run vid1Show again. What happens? You can use the DSS hardware to control where your video displays on the screen. Now you can't see the bmp file. Change this line echo 0  > $mgr0/alpha_blending_enabled to this echo 1  > $mgr0/alpha_blending_enabled and rerun vid1Show. You have now make the video buffer to appear behind the graphics buffer. Notice the video show through in the places in the bmp file that you transparent. Play around with it.

Try this: $ cd .. $ ./vid2Show $ cd lab07c_video_playback $ ./videoThru_DEBUG.Beagle Now we are using the hardware to display the video buffer in two places. You can turn things off with: $ cd .. $ ./resetVideo $ cd lab07c_video_playback

Part d - Video Loopthru
The goal of Part d is to combine the recorder (Part b) and playback (Part c) into a video loopthru application.
 * Hey – YOU get to do this yourself (no more inspection stuff…).


 * 1) Answer a few questions about the big picture.
 * 2) Copy files from Part c (playback) to Part d (loopthru).
 * 3) Add video input files from Part b (record) to Part d (loopthru).
 * 4) Make code modifications to stitch the record to the playback.
 * 5) Build, run. Result: video is captured (v4L2) and then displayed (FBdev) with your OSD.

In this portion of the lab, you will combine Parts a (video capture) and b (video playback) into a single video loop-thru application.

In Part b, we recorded video from the web cam via the v4L2 input and placed it into a file (video.raw) – this used an fwrite command to write the video buffer to a file. In Part c, we did an fread of the video.raw file and sent that video to the display driver.

We now have the input (capture) application (Part b) and the output (display) application (Part c) that you will now combine into a single application (Part d). We’ll need to get rid of the “file reads/writes” and replace them with a memcpy operation to copy data from Capture driver buffers to Display driver buffers.

Before Starting
Before we start copying, cutting, and pasting files and code, let’s think about what must be done to get the loopthru lab to work.
 * In Part b, we used fwrite to PUT (write) the video data to the video.raw file. What two functions were used to GET (read) the video data from v4L2 driver and return the video buffer back to the driver once the application has recorded the data?


 * Similarly, in Part c, we used the functions listed below to PUT (write) the data to the FBdev driver. What function is used to GET (read) the video data?

In this lab exercise, we will start with the Part c files, then edit them to create the loopthru code. Based on this, generally what functions should be required for our while loop in the Part d Video Loopthru?

To summarize, the following lab procedure will take the _capture and _playback files and combine them into a loopthru example.

Now, go do it.