Buildroot:GSoC2016Ideas

2016 will be the third year of participation of Buildroot to the Google Summer of Code program. This year, we are proposing the following ideas.

= Mentors =

The following people will be your mentors. Each topic is assigned a mentor, who will be your referent and will follow you during the GSoC. However, all mentors will be available to help you.


 * Arnout Vandecappelle
 * Peter Korsgaard
 * Romain Naour
 * Thomas Petazzoni
 * thomas.petazzoni@free-electrons.com
 * kos_tom on Freenode
 * Yann E. MORIN
 * yann.morin.1998@free.fr
 * y_morin on Freenode

(Missing detailed contact information will be added shortly.)

= Suggestions for candidates =

If you want to apply, we recommend you to prove that you have some basic knowledge about Buildroot and open-source contributions:


 * Subscribe to the mailing list and come to the IRC channel
 * Post some patches to the mailing list. Here are some possible contribution ideas:
 * Look at some items in our TODO list at http://elinux.org/Buildroot#Todo_list, implement some of them, and send patches
 * Find some software component that is useful on embedded systems, create a Buildroot package for it, and send patches
 * Find some embedded hardware platform, create a Buildroot defconfig for it, and send patches
 * Look at some build failures in our autobuilders at http://autobuild.buildroot.org and send patches to fix them. Note that some build failures may be difficult to solve.
 * Look at the bug tracker at http://bugs.busybox.net and send patches to fix them. Some bugs may be difficult to investigate.

= Topics =

The following topics are suggested by the Buildroot developers; they are not sorted in any specific order, not even of preference. Feel free to propose your own ideas.

Reproducible builds
Mentor: Yann E. MORIN

Short: Ensure that two runs of Buildroot with the same configuration yield the same result.

Abstract

A very important feature of a build system is to provide reproducible builds. Reproducible builds means that, given the same inputs, multiple builds of the same source will generate identical output, even when they are executed on different machines, at different times, in different locations, by different users. It is then possible to assess and independently verify that the source code is indeed what was used to generate a binary. It is then also guaranteed that a developer build and an automatic production build yield the same results.

Description

We would like to assess how reproducible the builds made by Buildroot are, identify and fix all the sources for non-reproducibility. This will involve creating an infrastructure to run builds twice and compare the results, and provide tools to identify the reasons for the differences. Finally, when possible, fix the source of the differences.

Reproducible builds are a hot topic, there is an entire website dedicated to it.

Skills


 * Intermediate Embedded Linux knowledge (cross-compilation, ELF format...)
 * Knowledge in the Python scripting language (existing testing infrastructures are written in Python, so it makes sense to continue in Python)
 * Knowledge of tools like Jenkins would be a plus.

Testing infrastructure
Mentor: Thomas Petazzoni

Short: Improve upon the existing testing infrastructures, build-time and runtime tests.

Abstract

The Buildroot project uses automated testing to help validate the stability of the project. There are currently two types of automated testing:


 * random configurations are used to help validate that the millions of possible Buildroot configurations build correctly. This automated build testing has been running for about three years (results visibles at ),


 * regression testing of a set of known configurations is used to track the sample configurations for a set of commonly-available embedded boards (Beagle Bone, Raspberry Pi, Wandboard...). This regression testing has been running for a few months now as a test-bed and now reports build results every other days.

These testing infrastructures have helped improve the quality of Buildroot. However, the project would like to bring a number of improvements to the random configuration testing infrastructure.

Description

We would like to add runtime testing, i.e really boot the generated system under Qemu, and verify, for the different packages that are part of the system, that they at least minimally work. Examples:


 * if the Python interpreter has been selected to be part of the system, verify that we can run it, and run a simple Python test application on the target;
 * if a given network daemon has been selected to be part of the system, verify that it is running and functional;
 * if C++ support has been enabled, verify that we can build (on the build machine) a C++ program and then run it on the target.

This involves creating an infrastructure to write tests in Python, and then run them inside Qemu, and report the results in a database.

Skills


 * Basic Embedded Linux knowledge (cross-compilation, kernel configuration/build, etc.)
 * Knowledge of the Python scripting language, used for the development of the testing infrastructure.
 * Some Web development skills and/or knowledge of tools like Jenkins would be a plus.

Relocatable SDK
Mentor: Yann E. MORIN

Short: Make the generated toolchain and staging relocatable.

Abstract

Buildroot generates a toolchain (compiler, binutils, libc) that can be used for other projects, or as a basis for another buildroot configuration. However, at the moment, the toolchain has to be installed in a specific (configurable) location and it is not possible to move it. We would like to make it possible to move the toolchain to a different location. This will allow the user to make a tarball of the toolchain, and easily re-use it as an external buildroot toolchain.

Also, once Buildroot has compiled all the packages from the configuration, the sysroot of the toolchain also contains all the headers and libraries that were used to generate the system. Having a relocatable toolchain would make it easy to prepare an SDK and distribute it to build applications outside of Buildroot (e.g. by third parties, or sub-contractors...) without the need to always build the entire system from scratch just to develop a single application.

Description

We have a pretty good idea of what needs to be done, and for some steps patches have already been prepared. Other things still have to be started.
 * Make sure that all binaries and libraries built for the host are built with a rpath pointing to host/usr/lib. Normally, this should already be the case, but it's worth checking.
 * Change the rpath value to $ORIGIN/../lib instead of the current absolute path $(O)/host/usr/lib.
 * Modify the compiler wrapper program of external toolchains so that instead of using a fixed location for the compiler tools, it deduces their location in a relative manner from its current location.
 * Modify/patch pkg-config so that instead of having a fixed location for the PKG_CONFIG_PATH and PKG_CONFIG_SYSROOT_DIR, those are deduced from the location of the pkg-config binary. This will allow a pkg-config binary that has been moved to still operate properly, without having to set any environment variable.
 * Write a shell script, installed in host/usr/bin, which would munge the libtool .la files, the qmake.conf file and the CMake toolchain file to set the correct path. This script reads a file (can be host/usr/share/buildroot/location) which contains the original location of the SDK. This allows the script to do the right modifications on all the libtool, qmake.conf and cmake files. Once this is done, the script changes the host/usr/share/buildroot/location file so that it contains the new location.
 * Modify the external toolchain wrapper so that it bails out and warns the user if the directory it is executed in doesn't match the location of host/usr/share/buildroot/location. We haven't discussed how this could work with internal and external toolchains, though.

Skills
 * Intermediate Linux knowledge (toolchains, shared libraries, ...)
 * Good shell scripting knowledge (the existing scripts are in bash shell)
 * Previous experience with Buildroot is required

Top-level parallel build
Mentor: Arnout Vandecappelle

Short: Allow a full top-level, concurrent, parallel build of packages.

Abstract

Building a complete system from source can take a long time. Most packages can use parallel build (i.e. make -j), and Buildroot takes advantage of this to shorten the build time. However, Buildroot can only build one package at a time; parallelism is only used at the package level, which is not optimally making use of the available resources. In particular, the configure steps of most packages are sequential, and they can take quite a large part of the build time.

Description

We would like for Buildroot to be able to build in parallel at the top-level, i.e. be able to build different packages concurrently, to maximise the use of the available CPU resources and consequently decrease the build time. This will require changes deep in the core of Buildroot and how we store and use the generated artifacts (staging, host tools...). The dependencies between packages will have to be investigated, to detect hidden dependencies (i.e. implicit or optional dependencies) and make them explicit.

A prototype implementation was made earlier, but this was incomplete and several corner cases have to be verified.

Note: some interaction with the Reproducible builds topic may be required.

Skills


 * Intermediate Embedded Linux knowledge (basic cross-compilation, ...)
 * Good knowledge of GNU make (functions, delayed expansion, variable scope, ...)

Follow upstream updates and CVEs of packages
Mentor: Peter Korsgaard

Short: Provide infrastructure to easily follow version changes in upstream packages and track security issues.

Abstract

Buildroot can build a large collection of packages, currently over 1800. It is very important that the most up-to-date versions be used, to get new features, security and bug fixes.

However, keeping track of all the new version releases for all those packages is a tedious, if impossible task. Thus we may miss very critical updates or only get them late; also some packages seldom get updated in Buildroot.

Description

We would like to add an automated procedure to detect new versions of packages, so we can update them as soon as upstreams release new versions. This solution would have to deal with various upstream delivery means, like http/https, ftp, git, etc... and report what packages are not up-to-date, what the currently-packaged version is, what the latest version is and how old it is; listing all intermediate versions (if any) would also be nice. Generating this report could then be added as one of our Jenkins jobs and the report posted on the mailing list (e.g. every week...). In addition, this automated procedure would check whether any reported CVEs (security issues) apply to the version currently packaged in Buildroot for each package.

There have already been a few attempts to provide such a script, but they were far from being complete and readily useable. They can however serve as a starting point and be improved.

Skills


 * Basic Embedded Linux knowledge (cross-compilation...)
 * Knowledge of the Python scripting language (to match the existing helper infrastructures)
 * Basic knowledge of the common VCS and DVCS (svn, cvs, git, mercurial, bazaar...)

Support for LLVM
Mentor: Romain Naour

Short: Add support for building an LLVM/Clang-based cross-toolchain, and ensure as many packages as possible still build and run.

Abstract

Currently, Buildroot supports building toolchains based on the GNU tools, gcc, glibc and binutils. Alternatively, Buildroot has a list of known, pre-configured toolchains that are publicly available from various sources (Linaro, Mentor...) and supports custom toolchains, but still expects those toolchains to be based on the same GNU tools.

The same goes for the debugger: only gdb is supported.

Description

We would like to add support for building toolchains based on the LLVM framework:


 * Clang as the compiler
 * DragonEgg as a gcc plugin
 * LLDB as the debugger

Optionally, it would be interesting to look at using other components from LLVM:


 * libc++ as the C++ standard library
 * lld as the linker

Skills


 * Advanced Embedded Linux knowledge (cross-compilation)
 * Good knowledge of GNU make (functions, delayed expansion, variable scope, ...)
 * Previous experience with Buildroot is required; experience with another buildsystem is a plus

Support new languages and complete existing ones
Mentor: Arnout Vandecapelle

Short: Add support for new languages (Go, Rust...).

Abstract

Buildroot supports some languages natively supported by gcc: C, C++ and Fortran. Those languages cover the large majority of software commonly used for embedded systems. Still, some commonly-used languages like Objective C/C++, Java, Ada are not supported.

Additionally, there is support for some of the most prominent scripting languages: shell (obviously), lua, nodejs, perl, PHP, Python 2 and 3, Ruby... While for some of them, we have a pretty good support and packages for lots of additional modules (Lua, Perl, PHP, Python...), some of them are not as featureful (nodejs, Ruby...)

Also, quite a number of interesting languages are currently not supported in Buildroot, like Go, Rust...

Description

We would like to enhance the support of currently-packaged languages, and add support for new languages. When it makes sense, we would also like to add infrastructures to manage those new languages, like we currently have for Python, Perl, Lua...


 * nodejs: currently, nodejs modules are bulk-downloaded and bulk-installed in the target without any kind of infrastructure. It would be great to add an infrastructure, not unlike the Perl/CPAN infra, to ease writing packages for NPM-based packages.


 * Ruby: apart form the Ruby interpreter, we do not have anything Ruby-related packaged in Buildroot. Maybe it would be interesting to add support for installing Gems.


 * Go: we do not have a Go cross-compiler (neither golang, nor gcc). There is a pending patch to add the golang compiler and a set of patches to add a golang infrastrucutre. Both still need some work (but the golang package is pretty much advanced).


 * Other languages to identify and see if it makes sense to have in Buildroot.

Skills


 * Basic Embedded Linux knowledge (cross-compilation...)
 * Basic to Intermediate knowledge of at least one or two languages above
 * Intermediate to Advanced knowledge of GNU make (functions, delayed expansion, variable scope, ...)
 * Previous experience with Buildroot is required; experience with another buildsystem is a plus