Buildroot:Top Level Parallel Build
This page documents the effort to support top-level parallel build in Buildroot.
The core support has been submitted in its v7 iteration at http://patchwork.ozlabs.org/project/buildroot/list/?series=83747. However, this core support does not fix a number of package-specific issues that arise with per-package directory support, and those fixes will be submitted separately.
This patch series is also visible at https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v7
Per-package directory fixes
All the fixes that we currently have are available at https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v7-work. This branch is regularly rebased.
- Issues with Python modules
- Need to avoid using Python Eggs for host Python setuptools packages. SUBMITTED at http://patchwork.ozlabs.org/project/buildroot/list/?series=83776
- Need to rewrite _sysconfigdata.py before building a Python module. PENDING until the core support is merged. It is fixed by package/pkg-python: fix for per-package directories
- Issues with Meson packages
- Need to use per-package Meson cross-compilation.conf file. This was anyway also needed to have per-package CFLAGS/LDFLAGS. SUBMITTED at http://patchwork.ozlabs.org/patch/1005880/
- Issues with apr/apr-util/apache packages
- These issues are fixed by a number of fixups. PENDING until the core support is merged. It is fixed by package/apr-util: fix build with per-package directories and package/apache: fix build with per-package directory support in our branch.
- Issue with the owfs package
- Needs a _sysconfigdata.py fixup. PENDING until the core support is merged. It is fixed by package/owfs: fixup Python sysconfigdata for per-package directories in our branch.
- Issue with the cargo package
- Needs to pass the appropriate linker flags. 'SUBMITTED at http://patchwork.ozlabs.org/patch/1019168/.
Known remaining issues
- The Qt5 packages are broken with per-package directory support. See the thread at http://lists.busybox.net/pipermail/buildroot/2018-December/237727.html for details.
- The build-time.log file is appended concurrently by multiple packages, probably not good.
- The package-file-list*.txt files are appended concurrently by multiple packages, probably not good.
- Some kconfig packages do not have a proper dependency on ccache if ccache support is enabled. This is however not strictly speaking a per-package directory issue.
To test top-level parallel-build:
- Grab the branch at https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v7-work
- Enable BR2_PER_PACKAGE_DIRECTORIES in your Buildroot configuration
- Run your build with make -jX
Note that the output will be garbled, as the output of multiple packages building in parallel will be mixed together. You can decide to build with make -jX --output-sync=target to have a cleaner output.
To visualize how well (or not so well) your build is parallelized, you can run at the end of a clean build:
- make graph-build-time
And look at output/graphs/build.timeline.pdf