Difference between revisions of "Buildroot:Top Level Parallel Build"

From eLinux.org
Jump to: navigation, search
(Per-package directory fixes)
 
(3 intermediate revisions by the same user not shown)
Line 19: Line 19:
 
* Issue with the owfs package
 
* Issue with the owfs package
 
** Needs a _sysconfigdata.py fixup. '''PENDING''' until the core support is merged. It is fixed by ''package/owfs: fixup Python sysconfigdata for per-package directories'' in our branch.
 
** Needs a _sysconfigdata.py fixup. '''PENDING''' until the core support is merged. It is fixed by ''package/owfs: fixup Python sysconfigdata for per-package directories'' in our branch.
* Issue with the Qt5 packages
 
  
 
= Known remaining issues =
 
= Known remaining issues =
Line 31: Line 30:
  
 
To test top-level parallel-build:
 
To test top-level parallel-build:
# Grab the branch at https://github.com/tpetazzoni/buildroot/tree/ppsh-v7-work
+
# Grab the current master branch
 
# Enable BR2_PER_PACKAGE_DIRECTORIES in your Buildroot configuration
 
# Enable BR2_PER_PACKAGE_DIRECTORIES in your Buildroot configuration
 
# Run your build with ''make -jX''
 
# Run your build with ''make -jX''
  
 
Note that the output will be garbled, as the output of multiple packages building in parallel will be mixed together. You can decide to build with ''make -jX --output-sync=target'' to have a cleaner output.
 
Note that the output will be garbled, as the output of multiple packages building in parallel will be mixed together. You can decide to build with ''make -jX --output-sync=target'' to have a cleaner output.
 +
 +
= Autobuilder testing =
 +
 +
The autobuilders are already testing ''BR2_PER_PACKAGE_DIRECTORIES'' and top-level parallel build:
 +
* 1 build out of 15 build is built with ''BR2_PER_PACKAGE_DIRECTORIES''
 +
* When ''BR2_PER_PACKAGE_DIRECTORIES=y'', one out of two builds is built in top-level parallel build mode
 +
 +
The build results with ''BR2_PER_PACKAGE_DIRECTORIES=y'' is available at http://autobuild.buildroot.net/index.php?symbols%5BBR2_PER_PACKAGE_DIRECTORIES%5D=y
  
 
= Graphing =
 
= Graphing =
 +
 +
This is only available in the ''ppsh-v8-work'' branch mentioned above, and not in master.
  
 
To visualize how well (or not so well) your build is parallelized, you can run at the end of a clean build:
 
To visualize how well (or not so well) your build is parallelized, you can run at the end of a clean build:
 
# make graph-build-time
 
# make graph-build-time
 
# evince output/graphs/build.timeline.pdf
 
# evince output/graphs/build.timeline.pdf

Latest revision as of 04:58, 3 February 2020

This page documents the effort to support top-level parallel build in Buildroot.

Core support

The core support for top-level parallel build was merged, is currently in master, and will be part of the 2020.02 release. What was merged is essentially:

  • Per-package directory support, through the BR2_PER_PACKAGE_DIRECTORIES option
  • Ability to do top-level parallel build using make -jX, when BR2_PER_PACKAGE_DIRECTORIES=y

Per-package directory fixes

All the fixes that we currently have are available at https://github.com/tpetazzoni/buildroot/tree/ppsh-v8-work. This branch is regularly rebased.

  • Issues with Python modules
    • Need to rewrite _sysconfigdata.py before building a Python module. PENDING until the core support is merged. It is fixed by package/pkg-python: fix for per-package directories
  • Issues with Meson packages
  • Issues with apr/apr-util/apache packages
    • These issues are fixed by a number of fixups. PENDING until the core support is merged. It is fixed by package/apr-util: fix build with per-package directories and package/apache: fix build with per-package directory support in our branch.
  • Issue with the owfs package
    • Needs a _sysconfigdata.py fixup. PENDING until the core support is merged. It is fixed by package/owfs: fixup Python sysconfigdata for per-package directories in our branch.

Known remaining issues

  • The Qt5 packages are broken with per-package directory support. See the thread at http://lists.busybox.net/pipermail/buildroot/2018-December/237727.html for details.
  • The build-time.log file is appended concurrently by multiple packages, probably not good.
  • The package-file-list*.txt files are appended concurrently by multiple packages, probably not good.
  • Some kconfig packages do not have a proper dependency on ccache if ccache support is enabled. This is however not strictly speaking a per-package directory issue.

Testing

To test top-level parallel-build:

  1. Grab the current master branch
  2. Enable BR2_PER_PACKAGE_DIRECTORIES in your Buildroot configuration
  3. Run your build with make -jX

Note that the output will be garbled, as the output of multiple packages building in parallel will be mixed together. You can decide to build with make -jX --output-sync=target to have a cleaner output.

Autobuilder testing

The autobuilders are already testing BR2_PER_PACKAGE_DIRECTORIES and top-level parallel build:

  • 1 build out of 15 build is built with BR2_PER_PACKAGE_DIRECTORIES
  • When BR2_PER_PACKAGE_DIRECTORIES=y, one out of two builds is built in top-level parallel build mode

The build results with BR2_PER_PACKAGE_DIRECTORIES=y is available at http://autobuild.buildroot.net/index.php?symbols%5BBR2_PER_PACKAGE_DIRECTORIES%5D=y

Graphing

This is only available in the ppsh-v8-work branch mentioned above, and not in master.

To visualize how well (or not so well) your build is parallelized, you can run at the end of a clean build:

  1. make graph-build-time
  2. evince output/graphs/build.timeline.pdf