Difference between revisions of "Kernel build bundle"
(initialize page, and add details about bootlin's 'custom test tool' project) |
(→kernel build system) |
||
| (5 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | This page describes a standard | + | This page describes a proposed standard for a "kernel build bundle". |
| − | This is used to describe the build artifacts from a kernel build, that can be referenced and passed | + | This is used to describe the build artifacts from a kernel build, that can be |
| − | to other tools - mainly a provisioning tool, to install a kernel into a test system. | + | referenced and passed to other tools - mainly a provisioning tool, to install |
| + | a kernel into a test system. | ||
This is not to be confused with boot standards, although those are related. | This is not to be confused with boot standards, although those are related. | ||
| − | It is anticipated that a kernel build system will produce a kernel build bundle (or a metadata file | + | It is anticipated that a kernel build system will produce a kernel build bundle |
| − | that references the kernel build artifacts), and that a test framework might use this to actually | + | (or a metadata file that references the kernel build artifacts), and that a test |
| − | provision a system. | + | framework might use this to actually provision a system. |
= pre-existing work and standards = | = pre-existing work and standards = | ||
== kernel build systems == | == kernel build systems == | ||
=== kernel build system === | === kernel build system === | ||
| − | + | The Linux kernel has a 'make install' target, that will populate a directory | |
| + | with artifacts from a build (or run a script to do so) | ||
| + | |||
| + | The logic for this is implement in the Makefiles scattered throughout the | ||
| + | kernel source tree. | ||
| + | |||
| + | By default, 'make install' will run the script install.sh in the respective | ||
| + | boot directory for the configured architecture. That is /arch/x86/boot/install.sh, etc. | ||
| + | |||
| + | [should document this more] | ||
=== kernelci === | === kernelci === | ||
| Line 27: | Line 37: | ||
It also has a modules.tar.xz file in the top directory. | It also has a modules.tar.xz file in the top directory. | ||
| + | |||
| + | ==== Bundle json file ==== | ||
| + | Here is a "bundle file: | ||
| + | https://s3.amazonaws.com/storage.lkft.org/rootfs/oe-sumo/20211006/juno/bundle.json | ||
| + | It has the following fields: | ||
| + | * kernel_config - name of kernel config file (that was used for this build | ||
| + | * dtb - name of the dtb file | ||
| + | * kernel - name of the kernel image file for this bundle | ||
| + | * rootfs - names of possible rootfs image files for this bundle | ||
| + | ** ext4gz - name of the ext4.gz file for the rootfs | ||
| + | ** tarxz - name of the tar.xz file for the rootfs | ||
| + | * modules - name of the archive file containing the modules | ||
| + | * modules_compression - type of compression for the mdoules file (eg. 'targz') | ||
| + | |||
| + | Daniel Diaz provided a link to a tool that takes an online directory, and creates | ||
| + | a bundle of files that can be used to install the kernel. It's called 'dir2bundle' | ||
| + | and it's here: | ||
| + | https://gitlab.com/Linaro/lkft/rootfs/dir2bundle | ||
| + | |||
| + | According to the description, 'dir2bundle' reads the contents of an images/ directory and | ||
| + | create a bundle.json file. It can read the contents of an online directory, or a | ||
| + | local directory, to create the file. | ||
=== tuxmake === | === tuxmake === | ||
| − | Tuxmake is a tool to | + | Tuxmake is a tool to handle kernel builds in a reproducible way. |
| + | |||
| + | See https://www.tuxmake.org/metadata/ for an (incomplete) description of their metadata | ||
| + | Here is an online directory with build results. But this build didn't work, so the build | ||
| + | artifacts are not there. | ||
| + | |||
| + | https://builds.tuxbuild.com/1wK6XBsWrtdspwuwyXX9cocbnXR/ | ||
== kernel installation systems == | == kernel installation systems == | ||
| Line 44: | Line 82: | ||
== tools that pass data around == | == tools that pass data around == | ||
| + | === bootline 'custom test tools' === | ||
Bootline has a tool called "custom_tests_tool" that can send the materials for | Bootline has a tool called "custom_tests_tool" that can send the materials for | ||
kernelci job to a server: | kernelci job to a server: | ||
Revision as of 13:31, 21 October 2021
This page describes a proposed standard for a "kernel build bundle".
This is used to describe the build artifacts from a kernel build, that can be referenced and passed to other tools - mainly a provisioning tool, to install a kernel into a test system.
This is not to be confused with boot standards, although those are related.
It is anticipated that a kernel build system will produce a kernel build bundle (or a metadata file that references the kernel build artifacts), and that a test framework might use this to actually provision a system.
Contents
pre-existing work and standards
kernel build systems
kernel build system
The Linux kernel has a 'make install' target, that will populate a directory with artifacts from a build (or run a script to do so)
The logic for this is implement in the Makefiles scattered throughout the kernel source tree.
By default, 'make install' will run the script install.sh in the respective boot directory for the configured architecture. That is /arch/x86/boot/install.sh, etc.
[should document this more]
kernelci
The kernelci system builds lots of kernels from mainline trees, with lots of different configurations.
Here is an example output manifest from one build: https://storage.kernelci.org/broonie-regulator/for-next/v5.15-rc1-lava-bisection-9108-pass/arm64/defconfig/gcc-8/artifacts.json
This has a description of information for this online directory: https://storage.kernelci.org/broonie-regulator/for-next/v5.15-rc1-lava-bisection-9108-pass/arm64/defconfig/gcc-8
which has sub-directories: config, dtbs, kernel, and logs
It also has a modules.tar.xz file in the top directory.
Bundle json file
Here is a "bundle file: https://s3.amazonaws.com/storage.lkft.org/rootfs/oe-sumo/20211006/juno/bundle.json It has the following fields:
- kernel_config - name of kernel config file (that was used for this build
- dtb - name of the dtb file
- kernel - name of the kernel image file for this bundle
- rootfs - names of possible rootfs image files for this bundle
- ext4gz - name of the ext4.gz file for the rootfs
- tarxz - name of the tar.xz file for the rootfs
- modules - name of the archive file containing the modules
- modules_compression - type of compression for the mdoules file (eg. 'targz')
Daniel Diaz provided a link to a tool that takes an online directory, and creates a bundle of files that can be used to install the kernel. It's called 'dir2bundle' and it's here: https://gitlab.com/Linaro/lkft/rootfs/dir2bundle
According to the description, 'dir2bundle' reads the contents of an images/ directory and create a bundle.json file. It can read the contents of an online directory, or a local directory, to create the file.
tuxmake
Tuxmake is a tool to handle kernel builds in a reproducible way.
See https://www.tuxmake.org/metadata/ for an (incomplete) description of their metadata Here is an online directory with build results. But this build didn't work, so the build artifacts are not there.
https://builds.tuxbuild.com/1wK6XBsWrtdspwuwyXX9cocbnXR/
kernel installation systems
There are multiple ways of installing a kernel so that it will boot on a board. Some of these installating methods are persistent (e.g. installed to a filesystem on the device, or a partition on the device), and some are ephemeral. Some examples of this later category are placing the kernel in a tftp directory on a host, or transferring a kernel into ram and executing it using adb.
- tftp boot
- etherboot
- /boot directory on filesystem on device (used by grub)
- in it's own flash partition
- in it's own disk partition
tools that pass data around
bootline 'custom test tools'
Bootline has a tool called "custom_tests_tool" that can send the materials for kernelci job to a server:
See this tool: https://github.com/bootlin/custom_tests_tool/blob/master/ctt.py
Some notes:
- CTTLauncher instance creates an 'artifacts' map, which has:
- kernel
- dtb
- modules
- rootfs
For any files mentioned in the _cfg, the launch() method can send the file to the REMOTE_ROOT, and change the reference to be local-relative (prepended with 'file://'), before submitting the job.
