Yocto Project Introduction
This is an introduction about the Yocto Project, written from the point of view of someone somewhat familiar with embedded Linux development and embedded Linux distribution maintenance. In my own investigation of the Yocto Project, I found that there were a few key concepts that I didn't find discussed or presented anywhere. I'd like to fix that with this page. The purpose of this page is to give a broad overview of how the project technically works, so that the learning curve for people approaching the Yocto Project for the first time is lessened.
The big picture
The Yocto Project is a collection of tools and meta-data (defined in a bit) that allows a developer to build their own custom distribution of Linux for their embedded platform. This could be a developer at a semi-conductor company, who wishes to develop board support for one of their hardware platforms, or it could be an independent developer writing a complete software stack for a product they are making. It could also be a group of engineers developing a distribution for use in multiple devices or products -- such as an embedded Linux distribution company, or the "systems" team at a company that produces multiple embedded Linux products.
The main parts
The main parts of the Yocto Project are the build system, the package meta-data, and the developer tools. The build system uses a tool called "bitbake" to process the meta-data and produce a complete Linux distribution. By design, the build system produces not just the software that will run on the target, but also the development tools used to build that software. It basically starts completely from scratch, building all the tools needed construct the software, and then using those to build the kernel, libraries, and programs that comprise a Linux distribution. Finally, it prepares the resulting software into appropriate bundles (including packages, images, or both) for deployment to the target device and in preparation for application development and debugging. The Yocto Project also includes various additional tools used to develop embedded Linux or applications on top of it. This includes things such as emulators, IDEs and host/target (cross) agents and debug tools.
Let's start by describing some of the concepts of the build system...
The primary tool used in the build system is called 'bitbake'. Bitbake has a user-manual at http://docs.openembedded.org/bitbake/html/ . (As of August, 2012, this document appears to be a bit dated, and missing a few items of importance.) Basically, bitbake can be thought of as "make" on steroids. It basically performs the same type of functionality as make - which is: determining the actions to perform based on 1) what the user requests at the command line, 2) the project data and 3) the existing build state, and then performing those actions.
Bitbake uses it's own new syntax for expressing:
- the tasks to perform
- the relationships (dependencies) between those tasks
- the variables that control how the tasks are performed
- the actual build instructions (e.g. compiler commands, linker commands, packaging commands, etc.)
Bitbake differs from 'make' in several key ways. The first is that it has a global view of the task list for a distribution. That is, it reads the entire set of files related to the distribution, and determines the global task list for a particular high-level build operation. This is considerably different from 'make', which processes just a single Makefile at a time. (Admittedly, 'make' can be made to work with extremely large projects, using complex include schemes and nested invocations. However, even these types of system are rarely used for something as complex as a complete Linux distribution build.)
Another apparent difference is that the syntax of the files that bitbake processes allows for a very high degree of flexibility in defining the tasks that should be performed and the variables that control the build process and output.
[note to consider: maybe too much flexibility?]
A variety of mechanisms (described in the bitbake manual) are used to control what operations are performed and what variables are used to control them. bitbake supports inheritance mechanisms, to allow for a class-like definition of common operations which can be customized for specific situations. It also supports the conditional definition of new tasks, and has the ability to customize or eliminate (mask) tasks based on variables computed during the build.
Bitbake is written in Python, and some aspects of the build system can be written using short Python snippets as well.
A key element of the Yocto Project is the meta-data which is used to construct a Linux distribution.
Meta-data refers to the build instructions themselves, as well as the data used to control what things get built and to affect how they are built. The meta-data also includes commands and data used to indicate what versions of software are used, where they are obtained from. The meta-data also consist of changes or additions to the software itself (patches or auxiliary files), used to fix bugs or customize the software for use in a particular situation.
The build instructions consists of commands to execute the compiler, linker, archiver, packaging tools and other programs.
The Yocto Project provides the build tool (bitbake) and meta-data itself, for a few distributions. Much of this meta-data can be re-used when someone is building their own distribution. The Yocto Project is related to the OpenEmbedded project, where the bitbake tool, much of the meta-data, and many of the meta-data concepts originated.
Each software component on the system (such as an individual program) has associated with it one or more files to express it's meta-data (dependencies, patches, build instructions). A top-level (usually single) file that defines the 'tasks' for the software is provided in a ".bb" (bitbake) file. This is referred to as that component's "recipe" file. This file may be terse, as the system allows for inheritance and inclusion. Class files (.bbclass) are used to express the meta-data for commonly-used types of build and packaging operations. These files are 'inherit'ed into a recipe. Include files (.inc) are sometimes used to provide a common set of definitions, which can be customized for a particular version of the software. These files are included into a recipe using a 'require' command. Finally, patches and auxiliary files may be associated with recipe. These files can be referenced by the build instructions (eg. applied to the software after fetching it and before compiling it in the case of patches) of the recipe.
A set of bitbake files that are related to a particular feature area can be organized into a "layer". This represents a collection of software or build tasks that can included into an overall distribution.
Finally, a user selects the individual bitbake files to include, or sets of files from different layers, by referencing them in their local configuration (conf/bblayers.conf) and expressing build controls, also in their local configuration (conf/local.conf).