Buildroot:DeveloperDaysFOSDEM2017

= Buildroot Developers Meeting, 6-7 February 2017, Brussels =

Location and date
The Buildroot community had a meeting on February 6th and 7th 2017, for Buildroot developers and contributors. This meeting is a mixture of discussion and hacking session around the Buildroot project. The meeting took place in Google offices, located Chaussée d'Etterbeek 180, 1040 Brussels, very close to the Schuman metro station. Lunch was free of charge, courtesy of Google.

Sponsors
We would like to thank our sponsors:
 * Google, providing the meeting location, with Internet connection, but also free lunch and refreshments for the meeting participants.
 * Mind is the Embedded Software division of Essensium, which provides consultancy and services specifically in the field of Linux and Open Source SW for Embedded Systems. Mind offered the Monday dinner to the participants of the meeting.

Participants

 * 1) Thomas Petazzoni
 * 2) Yann E. MORIN
 * 3) Samuel Martin
 * 4) Peter Korsgaard
 * 5) Luca Ceresoli
 * 6) Romain Naour
 * 7) Maxime Hadjinlian
 * 8) Julien Boibessot
 * 9) Vicente Olivert Riera
 * 10) Thomas De Schampheleire
 * 11) Gabe Evans (Tuesday only)
 * 12) Randy Shifflin (Tuesday only)
 * 13) Hans Mortier
 * 14) Arnout Vandecappelle
 * 15) Jan Heylen (Monday only)

Google Summer of Code
We have no idea what criteria Google uses to decide to reject our applications. Maybe it's because our two previous GSoC students didn't perform well and we didn't handle it well either.

We recycle the ideas of Buildroot:GSoC2016Ideas. Most topics are still applicable. The edits are applied directly to Buildroot:GSoC2017Ideas.

For Reproducible Builds, we need to synchronise with Jeröme to check if it's useful and if he want to be a mentor.

The "new languages" topic still needs to be reviewed; Arnout will do this.

Host packages with no in-tree user
Two use cases: The core problem remains: we don't want to add too many host packages. However, host packages are needed for reproducible builds. So people will need to add some kind of chroot/container. Maybe instead we should allow more host packages with a prompt. But what with things like python-lxml, which is a random python module that may be needed for an out-of-tree package; it's really weird to see something like that in the menu. For that one, however, installing it on your system doesn't help because the system python packages are NOT used by $(HOST_DIR)/usr/bin/python. Maybe for these it's not even so important that they are tested in the autobuilder - almost nobody is going to use it anyway. However, in that case it's not very useful to have it anyway. Especially because it almost certainly break at some point.
 * Packages on which an out-of-tree package depends (cfr. a lot of SELinux things). These don't need a Config.in.host prompt, but then they won't be triggered by autobuilders. A workaround for this is to add 'bool "prompt" if BR2_ENABLE_TEST_HOST_PACKAGES', where BR2_ENABLE_TEST_HOST_PACKAGES is enabled through an environment variable by the autobuilders.
 * Packages used by out-of-tree post-build/image scripts. These really need a prompt otherwise they can't be used.

We discussed the option of just shipping a Buildroot docker or other container and put any host package in there. But that is really not the direction we want to go. We want to be able to take any Linux machine, extract the tarball and do 'make'.

Conclusions:
 * We certainly need a Config.in.host for all host packages.
 * Once that is done, we can add the BR2_ENABLE_TEST_HOST_PACKAGES hack.
 * We loosen up a little on accepting host tools with prompts, at least if there is a corresponding target package.

Two-toolchain support
Aarch64 does not make it possible to combine a 64-bit and 32-bit toolchain (on amd64 this is possible with -m32 and -m64). It is thus not possible to create a 64-bit kernel combined with 32-bit userland. To support this, Buildroot needs to 'know' about two separate toolchains, and allow some packages to build with one and some with the other.

It gets worse, because some applications have to use the 64-bit toolchain in order to use the ARMv8 SIMD instructions. So this requires a multilib rootfs.

So there are two things that would be needed: One possible solution is to use the ilp32 ABI, which is a mixed 32/64-bit ABI similar to x32. This would allow the entire userspace to be built with the same toolchain, and the kernel can also use the same toolchain. However, it's not clear if upstream glibc and kernel are fully ready for that.
 * Supporting two toolchains, and allow packages to select this one they want to use.
 * Supporting a multilib rootfs.

The other solution is to use two separate configs with two toolchains and mix the rootfses. To do that well, the libraries would have to be copied around. The easiest is to just chroot.

clapack size difference in Debug vs Release cmake settings
When no BUILD_TYPE is set in cmake, the clapack library on MIPS is the smallest. When BUILD_TYPE is Release, however, the package becomes a lot larger (2-3 MB). After analysis by ThomasDS and Samuel, it turned out to be recently fixed by Samuel in the cmake infrastructure. The fixes are part of 2016.11.

Updating the DEVELOPERS file
It's actually not so convenient that the DEVELOPERS file is updated in a separate patch. The reason to do that was to make cherry-picking easier, but actually the risk of conflicts isn't that high and they're easy to solve. With a separate patch there are more patches to apply so more complexity. Also the documentation currently doesn't mention that it should be a separate patch. So let's leave the documentation alone and in the future to do what is asked in the documentation: update DEVELOPERS in the same patch.

opkg package generation
Nokia has a use case where the NOR is too small to contain the full system. So the files of some packages have to be removed from the rootfs and moved to a different output directory. This directory is then bundled as an opkg, put in NAND, and at runtime extracted over the initramfs. Is (part of) this solution upstreamable?

Partly it also comes down to adding per-package options. If we want to make things like this configurable, we would need for each package a Kconfig option to choose whether or not is should be an opkg. But also we would like to be able to specify per package static linking, debug/optimisation options, etc. This is not feasible in Kconfig.

Actually, anything can already be overridden in local.mk by putting override in front of it. But if you do this, local.mk becomes part of the Buildroot configuration so it should be under version control. Then it probably should still include a real-local.mk to allow non-version-controlled overrides... Basically, the necessary infra is there but it could be documented better.

It is also possible to add a Kconfig option with a list of packages that have to be overridden. This can be done for extracting opkg, static/dynamic, debug/release, ....

kexec / kdump for kernel crash dumps
kdump is a tool that allows you to run a kexec to a second kernel when an OOPS/panic occurs. This kernel can do things like saving the kernel core or even the entire memory. You typically want a different kernel build for the second kernel and include an initramfs in it, and it needs to run from a different physical address. So you need to build the kernel twice. Is it possible to do this from a single defconfig? Today that's not possible, you just make two configs and build twice, perhaps calling the second build from the post-image script.

Stuff like that is nice to be in the buildroot repository even if it is not usable directly from Kconfig. It is basically a collection of scripts and Kconfig fragments that are glued together. We need to make sure that such auxiliary tools are:
 * findable;
 * tested;
 * documented.

See also - a wrapper around Buildroot that manages multiple configurations. It could be useful to refer to this project from the documentation.

using quilt to manage patches
Basically this is not something that will be part of Buildroot. Perhaps we could add a support script that makes it easier, but it is actually a lot simpler to just explain in the manual how to run a few shell commands that do it. This way the developer has full flexibility to do what he wants. It's not complicated after all.

Security and bugfix releases
This cycle Peter made several bugfix/security releases (2016.11.1 and 2016.11.2), with roughly one change per day. Peter wants to release 2016.11.3 now. All of this has just been cherry-picking version bumps with security issues, also a few bugfixes but not autobuild failures. There is currently no testing at all of this branch. If we continue doing this for a longer time it's more and more likely that things will break on the way. If we do start autobuilder testing of the stable branch, we would also need to use a different set of toolchains - e.g. when the default switches to gcc6, the stable branch should still be mainly tested with gcc5. Also we don't need to test the stable branch with all exotic toolchains, e.g. just ARM, x86, MIPS is sufficient, musl probably not needed.

The main purpose of the maintenance branch is that people get notified of fixes. Maybe there isn't as much need to do build testing of that branch. Maybe what Peter is doing now is enough. If we see that people encounter problems, we can react to that.

We also need to decide what we consider an LTS. Either we keep 2016.11 as the LTS, or we now announce the 2017.02 as an LTS. Then during one year we have one stable and one master branch. Since we have no particular preference either way, let's take 2017.02 so that we are using something with "2017" in the name all through the year. Bumping LTS once per year fits with Pengutronix' suggestion that a company should look every year at packages falling out of maintenance and bump those.

It could be a good idea to track in the CHANGES file of stable when packages are vulnerable to some CVE. In some cases CVEs pop up for a Buildroot package where the fix is just too difficult. E.g. our webkit-gtk version is not maintained any more so there will be no simple version bump to fix it, and even a patch will be hard to backport. In that case, reporting the vulnerability is better than ignoring it.

Also other fixes than security fixes might be relevant. Some projects distinguish bugfix releases from major releases, so it would make sense to do the bugfix bumps in the stable branch even if they don't have a security implication.

Peter uses to track security issues. It's still a lot of manual work though to track the information. Something like that would be very nice to have for Buildroot as well, as the way to report security issues, but that looks like a lot of work.

Conclusions:
 * We use 2017.02 as the LTS
 * We announce that 2017.02 will be maintained for a year.
 * We phrase it carefully that we will try, and that there will be "regular releases" i.e. not commit to a month.
 * We make sure that this announcement appears on lwn.net.
 * We do consider non-security bugfixes as applicable, on a case-by-case basis.
 * We document a process to propose patches for stable. E.g. the command that Peter uses to pick up patches from master. And extend it with something like 'Branches: 2017.02.x'.

status check: per-package staging / target and top-level parallel build
Gustavo is going to work on this this year.

Test suite
Maxime and Thomas worked on it, Luca reviewed it a little. What is still missing is documentation on it (both for using it and writing tests, and also how it works internally). It's just enough to get a foot in the door. Luca is willing to write some documentation while reviewing. It is something that can be applied even if it is not perfect yet.

Arnout created a Gitlab-CI configuration for building the defconfigs. It will be easy to add the tests to that. Arnout has created an official Buildroot repo on gitlab: so this will build every time the Buildroot repo is synced from git.buildroot.org, which is once per hour. To avoid running it all the time, the defconfigs will be limited to run on explicit trigger and the API will be used to trigger them daily.

Cupertino Miranda of Synopsis posted a web interface a while ago which visualises failed an succeeding packages. It wasn't great, because it is scraping the website instead of querying the database, it's written in Ruby, and the Ruby isn't very good either. We're also not sure to what extent the visualisation is really relevant for Buildroot. What kind of visualisation would be relevant for Buildroot? Something that is certainly missing now is knowing which packages succeed to build, we can also show failures. There is also no view of how things evolve over time, or a distribution over arch/libc/static/... Perhaps we can add a read-only SQL access and maybe even a web interface like redash. If people want to experiment, Thomas can execute queries on demand.

list of "big" patch series to discuss/review/work on

 * Reproducible builds: partially reviewed by Samuel
 * Git download method improvements: reviewed by Arnout. It is not (anymore) oriented towards "special refs" so no longer controversial.
 * Still some SELinux stuff: needs review
 * check-shlibs-deps: looks a bit dangerous, maybe not for 2017.02
 * Xtensa overlay stuff: looks very ugly, we'd like to reject it... but we have it in gcc and binutils as well, and with the xtensa tooling it's difficult to do it differently.
 * check-package: coding style is not OK, but is very useful so can be applied still for 2017.02.
 * Qt 5.8: can be applied with minor fixes so it is still in 2017.02.
 * External toolchain improvements from ThomasDS: reviewed orally, ThomasDS will send an updated version with minor details fixed. It's horrible but it makes sense.