Buildroot:DeveloperDaysELCE2017

Location and date
The Buildroot community organized a meeting on October 21st and 22nd 2017 in Prague (Czech Republic), for Buildroot developers and contributors. This meeting was a mixture of discussion and hacking session around the Buildroot project.

The meeting took place at the MacGyver Hackerspace, located Chaloupeckého 1916/5, 160 17 Praha 6-Břevnov-Praha 6. We would like to extend our gratitude to the MacGyver Hackerspace for housing us completely free of charge, and especially for our host Petr Barton who also assisted us with ordering food and who gave us an MQTT-driven display that reports the number of patches remaining in patchwork.

Sponsors
We would like to thank Mind, which is sponsoring the Sunday night dinner at Chabad Grill. Mind is the Embedded Software division of Essensium, which provides consultancy and services specifically in the field of Linux and Open Source SW for Embedded Systems.

Participants

 * 1) Thomas Petazzoni
 * 2) Arnout Vandecappelle
 * 3) Peter Korsgaard
 * 4) Yann E. MORIN
 * 5) Alexey Brodkin
 * 6) Erico Nunes
 * 7) Baruch Siach (Sunday only)
 * 8) Luca Ceresoli (Sunday late afternoon only)

Support Buildroot on alternative OSes (MacOS, Linux-on-Windows, xBSD)

 * If it works, fine. If small patches are needed (like we had before for xBSD), fine. If it doesn't work, use the Vagrantfile to run BR in a VM.
 * Alexey will give it a go on WSL and MacOS.

How do we handle glibc 2.26 patches?

 * glibc is NOT going to make point releases
 * Options for us
 * Adding individual patches
 * Not a good idea, difficult to be sure that we don't miss one.
 * Fetching patches from git
 * Cloning using git/http/https triggers a download (SSL) issue in gitlab when old Debian squeeze chroot is used.
 * Very big .mk file, we need to specify >53 patches
 * Use git download instead of release tarball
 * Very annoying that the version is an unreadable sha1 instead of 2.26.x. We can use 'git describe' output extended to 40 hash characters.
 * Output of `git describe` is not fetchable from a shallow clone, so we end up doing a full clone
 * When glibc is updated, we would normally update to the latest commit on the branch.
 * Automatic detection of when there is a new commit on the repo
 * We want things to be at least somewhat tested first - with a notification, it's likely that someone will just do a build test and nothing more.
 * We don't really want to bump the version 80 times per year.

Romain Naour already sent a patch switching to a commit in a github mirror (which is much faster than the sourceware repo).

Support for "language" package managers

 * Cfr. http://lists.busybox.net/pipermail/buildroot/2017-October/203779.html
 * There is not a single package manager that satisfies these requirements, even requirement 1.
 * Add these requirements and the above conclusion to the FAQ [Arnout]
 * We could go the NPM way for other package managers as well (i.e. "you're on your own").. If someone proposes something like that we'll accept it.
 * There is actually not a big problem with packaging individual modules e.g. in Python, so there is no big need for supporting package managers.
 * We could add a flag that if you use something like NPM, at the end a big warning is shown that the build is not reproducible, and also legal-info output has this warning.

Evaluation of LTS releases

 * Thomas wrote a blog post.
 * Nobody at the meeting actually uses these releases, but Peter has convinced people in his company to occasionally update to newer LTS point releases.
 * We also get occasional patches on the list specifically for LTS, so clearly some people are relying on it.
 * Another attendee's company will switch to an LTS sometime soon. They won't track the branch nor every point release, they'll just update when they have an issue. The mere fact that this branch is maintained is a good point.
 * When Peter stops maintaining 2017.02.x, the branch doesn't close so other people can still step up and cherry-pick patches on it. Also someone can step up and take commit access on this branch and who can make releases.
 * Peter will announce this option when he makes the last 2017.02.x release.
 * This person just needs to look at all the patches that go into 2018.02.x and decide if they need to be backported.
 * Do we need rules for pushing things to the LTS branch?
 * E.g. a for-stable tag in the commit message.
 * The committer can add this tag if the submitter hasn't done so.
 * Now there is a kind of tag: "security bump" in the commit title. But Peter backports a lot more than those.
 * We don't want to make the process too heavy.
 * Just replying on the mail works for Peter as well.
 * The ways to get something in the LTS branch should be documented [Alexey]:
 * "security bump" in the subject line
 * [PATCH 2017.02.x] in the subject line
 * Reply to the patch saying that is should be applied to the LTS branch
 * But first of all, a reference to LTS should be added to the manual.
 * What's the process? When there is an issue, do we cherry-pick a patch or bump the version?
 * Now it's a pragmatic approach: if it is easy to backport a patch or upstream maintains an LTS branch, then we use that, otherwise we just bump the version.
 * Point releases are technically not useful because they don't get any specific testing, but it makes it easier to convince management that this is something you should update to. It's also useful that there is a release mail sent out so people are reminded to update.
 * It would be useful to get the autobuilders working on LTS. Also the gitlab-ci
 * would require backporting the relevant patches
 * Maybe just wait for 2018.02.
 * We can now start doing autobuilders on the 2017.08 branch.
 * Reporting should be updated to mention the branch.
 * People in DEVELOPERS file are expected to also take care of the LTS branch. Perhaps this should be documented somewhere.
 * Tree-wide changes are extremely annoying for backporting, e.g. the license tag change. So tree-wide changes are ideally done in December-January.
 * It would also be nice to bump packages to their LTS branch just before XXXX.02.
 * So avoid bumping to a non-LTS version in the XXXX.02 release cycle.
 * Most packages don't have LTS branches anyway.
 * This is not a hard requirement, just a nice-to-have.
 * e.g. for nodeJS we're not using an LTS branch. But the current branch is going to become an LTS so that will resolve itself.

Documentation

 * Thomas called our documentation crappy
 * Main problem is the structuring: no good distinction between introductory material and reference material. E.g. the "adding packages" section is purely reference material.
 * The Free Electrons training material is a much better start as introductory material, after that the manual is a good reference manual.
 * We should add something like this to the manual itself, but nobody volunteers to do that.
 * Erico and Arnout OTOH think that the manual is good enough. They use it all the time as a reference to explain to people how things work.
 * Sometimes there are things you want to document and there is no good place to put them.

Release notes

 * Historically Peter just updates the CHANGES file close to the release.
 * When people add a new feature, it is nice if there is an update to the CHANGES file.
 * but in a separate patch, for cherry-picking
 * It's also OK if you mention it in the commit message, but it could be difficult to find a patch where this applies.
 * The committer can also do this directly.

Move packages tests to the corresponding package directory (side by side with the .mk)

 * Moving tests is OK, but it's more important to do other things
 * But if the patches are there to move it, we can do it.
 * Many tests however are not package tests, e.g. SELinux support. So those will stay in the test directory. Same for init system tests.
 * Where to put common classes. For Python e.g. it's possible to put it in the python package, but for the common part of init systems it is not.
 * Most important thing is to add more tests; these details can still be changed later.

Buildroot association

 * The bank account costs about 12 euro every 3 months. The money we have is only used for this. It will be empty in about 8 years.
 * Are we going to spend all our money funding the bank?
 * We can raise more money by having members, but then we need to give the members some reason to pay membership.
 * Backup of autobuild server - pay for storage
 * Domain name registration
 * CI services
 * Travel costs but that's a different scale
 * We could move sources.buildroot.org off from Peter's personal server, but we don't really want to pay for things that we can now get for free, even if the current situation is perhaps not ideal. Same about moving off osuosl.
 * Perhaps we should state goals depending on how much sponsorship we get.
 * We need to show people that we need money. The money should be used for more than what we're doing anyway already without money.
 * Web page that shows we have an association and what it gets used for.
 * Leaflet for the association and what will be done with it. If we get so much per year, we can do that.
 * CII best practices badge requires that there is no single point of failure for some things, e.g. ownership of domain name. The CII checklist is a good source to know which things we should take (better) care of. The association can be used to take ownership of these things.
 * Having multiple people having access to the core machines (e.g. autobuild.buildroot.org) would definitely be a good thing, but that doesn't mean we necessarily have to pay for it.
 * Membership is as much as you want but minimum 10 euro.
 * Registration is done by pushing a patch to the website that lists the member.
 * Arnout volunteers to write a webpage
 * Peter asks osuosl if it's feasible to host sources.buildroot.org and and autobuild.buildroot.org there. The association can buy for the HDD.
 * Thomas will set up a Paypal account so people outside the EU can easily pay.

Google Summer of Code

 * Let's skip GSoC this year?
 * Perhaps an interesting topic would be infrastructure tooling, e.g. backup of autobuilder, script that checks for upstream releases, script that pulls in things from patchwork and run check-package on them, ...
 * Would Google accept such a topic? Esp. since they didn't accept us at all the two previous times...
 * We can mostly reuse the topics from last year.
 * Reproducible builds
 * Testing infrastructure
 * Follow upstream updates and CVEs of packages
 * Support for LLVM

Fix gitlab

 * Gitlab mirroring feature doesn't work anymore (since 2 months now).
 * Apparently a known issue.
 * For now, do a "manual" push from a cron job on the a.b.o server [Thomas]
 * Done
 * Make gitlab CI more visible on BR webpage, manual, ...
 * Doesn't make sense until the pushing is fixed...

Top-level parallel build
Gustavoz' latest work is available at http://repo.or.cz/buildroot-gz.git/shortlog/refs/heads/pps3-next. This work mainly focuses on per-package staging. That concept will have to be extended to per-package SDK, i.e. including the host directory and also doing the (rpath) fixups that are now done in the sdk target.

One thing missing in Gustavoz' work is handling the case when two packages write the same file. The problem occurs when you have packages as follows: A -> B -> D, A -> C -> D. A creates a file foo, B overwrites that file with foo'. The PPS of D should then contain foo'. However, in Gustavoz' implementation, the PPS of D will be populated in the order listed in the dependencies, so first B, then C. The PPS of B contains foo', but the PPS of C then overwrites it with foo again. There are two solutions to this:


 * 1) Don't allow overwriting any file. We should check to what extent files are overwritten at the moment. We know for sure that busybox has a lot (in target), but we have no idea how often it happens in host and staging. We can detect overwrites by taking a hash of all files after staging-install, similar to what we do now for packages-file-list.txt. If we detect that a file gets overwritten, we bail out and say that parallel builds are not allowed for this config.
 * 2) Allow overwriting files. Then we need the PPS that results after staging-install to contain only the files that we created by that package. So we still need the hashing to find the files created by that package. Then, when creating the PPS for a package, we have to recursively go through all dependencies in a predictable way. It has to be recursive to satisfy dependency order. In the example above, it would mean we use first A, then B, then C to build the PPS of D. Since C does not contain foo, the end result will have the foo' from B. In this case, however, it is not detectable when both B and C overwrite foo. This is particularly annoying if packages append to a file: in the current situation, that works fine because the order in which things are done is predictable and it's all serial. But with top-level parallel build, we end up with a foo'1 and foo'2 which each have only a single append operation applied to them, so the end result will be missing one of the appends. This is (as far as we can see) not solvable. Therefore, this solution is not an option.

We should first find out how big the problem is. For that, Yann has sent a patch series that extends the packages-file-list.txt with the host and staging packages. This series now includes a patch that reports (without error) any duplicates in those lists. Then we can look at the successful autobuild results and check if there are any duplicates, and if so which ones.

Patchwork backlog
Patches that need discussion
 * minetest, supertux, and other games
 * Who is ever going to install all these games in a Buildroot system?
 * It's interesting to have some 3D game as a demo of GL integration.
 * If it's a simple package it doesn't hurt either.
 * Alexey will contribute Open Arena, which will be used as the Vivante demo for ARC.
 * minetest is actually simple enough, so let's merge it.
 * pppd: http://patchwork.ozlabs.org/patch/826889/ or http://patchwork.ozlabs.org/patch/806648/ ?
 * The latest patch is keeping backward compatibility so that one is applied.
 * openssl/libressl virtual package http://patchwork.ozlabs.org/project/buildroot/list/?series=8820
 * virtual package looks good
 * problem with the packages that need to be patched: most of them add something like '|| defined(LIBRESSL_VERSION_NUMBER)' to the conditions that check for older openssl versions. This is not really upstreamable, because when a later version of libressl does add those APIs, then the condition will have to change again and it will become pretty complicated. The problem is that libressl claims to be openssl 2.0.0, so any check that the openssl version > 1.1.0 will actually succeed. We consider this a bug in libressl, they really shouldn't claim to be openssl 2.0.0, they should claim to be 1.0.99 or something similar. The options for Buildroot:
 * Remove libressl entirely - is it really that useful? OpenSSL has improved a lot since heartbleed.
 * Take the patches as they are now and deal with the fallout when the package is bumped; too bad if libressl ever does support those APIs. In this case, we should certainly ask Adam to upstream the patches.
 * Patch libressl to declare version 1.0.99 instead of 2.0.0. That doesn't solve the problem if libressl ever does support those new APIs, since the old ones will still be used.
 * Instead of patching, let all of them depend on openssl until upstream has applied the patch.
 * Still a problem with this: in the current softether patch: if you change to libressl then softether silently disappears.
 * Introduce some kind of "REQUIRE_OPENSSL" that packages can select if they do require openssl (i.e. do not work with libressl)
 * Thomas implements this and applies the series.
 * Turns out that softether would be the only package that makes use of this option. All other packages don't select openssl but only use it optionally, they don't need this option, the condition just becomes LIBOPENSSL instead of OPENSSL. It's not worth to add this option for just this one package, so in the end we apply it as is.
 * Get feedback on minimal genimage Config.in options: http://patchwork.ozlabs.org/patch/805170/ and http://patchwork.ozlabs.org/patch/805171/ [Thomas]
 * Arnout doesn't oppose too strongly any more, but there were comments on the patches => Changes Requested
 * openjdk bootstrap (for kodi) http://patchwork.ozlabs.org/project/buildroot/list/?series=6229
 * Basically, it allows to build kodi on a host that doesn't have Java installed by downloading a Java binary from BLFS.
 * Is that really worth it?
 * Rejected
 * print dl hash: http://patchwork.ozlabs.org/patch/791357/
 * It is indeed annoying that you have to (manually) re-download and recalculate the hash
 * Actually, the most annoying thing is having to re-download.
 * The check-hash script already prints the new hash, but only if a hash existed already. So currently, if you bump a package, you can (manually) update the version in the hash file so that you get the new hash.
 * Still it's useful, so let's apply it and ask Gaël to also take a look at not removing the file if the hash check fails. The desired behaviour is already there in case the file already existed in the download dir, but not when it was just downloaded.
 * Handled the +- 10 oldest patches in patchwork

All in all, about 80 patches were processed.

Other work

 * Erico worked on grub2 (rework ARM support, rework grub2-tools package split)
 * Yann created a patch series to help finding packages that overwrite files in target, staging or host (see above).
 * Alexey and Erico started working on a package for 'piglit', a collection of tests for OpenGL.

Things we did not get around to
The following were scheduled for patchwork cleanup discussion but we did not get around to doing it:


 * Use host compiler to build extlinux: http://patchwork.ozlabs.org/patch/820225/
 * Custom linux logo: http://patchwork.ozlabs.org/project/buildroot/list/?series=3844
 * Realtek-hack patch for hostapd: http://patchwork.ozlabs.org/patch/813786/
 * arch: minimal required gcc version: http://patchwork.ozlabs.org/project/buildroot/list/?series=1206
 * genrandconfig: don't check certificates: http://patchwork.ozlabs.org/patch/809150/
 * software stacks: http://patchwork.ozlabs.org/project/buildroot/list/?series=7450
 * -mfused-add: https://patchwork.ozlabs.org/patch/800406/

The following work was planned to be done during the meeting but we didn't get around to do it:


 * EFI related stuff, building EFI firmware for testing with qemu
 * Supporting Secure Boot, e.g. required tools and option to sign grub and kernel binaries.
 * Finish and submit assorted unfinished patches [Erico]
 * out-of-tree per-package build [Yann]