Systemd Udev tuning
Introduction
This page will present various techniques to analyze and optimize the userspace startup sequence when using the systemd init system. A fairly recent version of systemd is assumed (e.g.: v256 was considered while writing this guide), but most of the techniques are applicable (or can be adapted) also to older versions.
Yocto HowTo
For most of the presented techniques, indications on how to implement the desired modifications in Yocto Project are reported.
Analyzing the init process
systemd offers a powerful tool to analyze its configuration and the way it impacts on the boot process: systemd-analyze.
The official documentation for systemd-analyze is very exhaustive and can be found here.
Yocto HowTo
The systemd-analyze tool is provided by a package having the same name, which is not installed automatically when using systemd as the init system. It can be thus added to an image using e.g.: IMAGE_INSTALL:append = " systemd-analyze".
Command: systemd-analyze
When called with no arguments, systemd-analyze shows a summary of the time required to boot, split between kernel and userspace. Moreover, the amount of time required to reach the default target (graphical.target in the example below) is reported.
root@hostname:~# systemd-analyze Startup finished in 11.166s (firmware) + 2.616s (loader) + 1.365s (kernel) + 3.406s (initrd) + 14.794s (userspace) = 33.348s graphical.target reached after 14.768s in userspace.
Things to consider when evaluating these numbers:
- All times depend on specific counters to be present and enabled; actual times might (or better, will) be higher, especially for the
(kernel)component. - The
(firmware)and(loader)components may not be present on all systems, as they depends on ACPI or UEFI support and may not be reported by hardware even if these two are in place. - The
(initrd)component won't be present if an initrd is not used. - The
(kernel)time includes the time required to load the init binary and its dependencies from the root filesystem and launching it - this is the reason why it is higher than the time the kernel itself reports when printingRun [...] as init process.
Command: systemd-analyze blame
The systemd-analyze blame command prints the time taken by the running units to start, excluding the ones with Type=simple (as systemd has no mean of measure how much they took). The output can be misleading, as times it reports may include the time spent waiting for other units to initialize, and shall thus be complemented with other measures (and a grain of salt).
root@hostname:~# systemd-analyze blame 7.979s systemd-networkd-wait-online.service 3.517s dev-mmcblk1p2.device 1.334s systemd-udev-trigger.service 1.297s systemd-timesyncd.service 1.034s systemd-resolved.service 848ms systemd-networkd.service 710ms dbus.service 625ms systemd-logind.service 597ms sshdgenkeys.service 583ms dev-hugepages.mount 568ms dev-mqueue.mount 557ms sys-kernel-debug.mount 541ms systemd-tmpfiles-setup.service 532ms tmp.mount 509ms kmod-static-nodes.service 502ms systemd-udevd.service 480ms modprobe@configfs.service 476ms systemd-random-seed.service 474ms modprobe@drm.service 456ms modprobe@fuse.service 409ms systemd-network-generator.service 387ms systemd-remount-fs.service 367ms proc-fs-nfsd.mount 342ms systemd-sysctl.service 329ms systemd-userdbd.service 328ms systemd-udev-load-credentials.service 297ms systemd-journald.service 285ms sys-fs-fuse-connections.mount 248ms sys-kernel-config.mount 219ms ip6tables.service 211ms var-volatile.mount 210ms iptables.service 191ms systemd-vconsole-setup.service 152ms systemd-tmpfiles-setup-dev-early.service 139ms systemd-journal-flush.service 123ms systemd-networkd-persistent-storage.service 112ms rpcbind.service 109ms systemd-update-utmp.service 99ms sshd.socket 77ms systemd-tmpfiles-setup-dev.service 63ms rpc-statd.service 57ms rpc-statd-notify.service
Command: systemd-analyze critical-chain $UNIT
The systemd-analyze critical-chain command, which shall be called with a unit name (e.g.: myapp.service), can be used to print the sequence of units that lead to the startup of the specified unit. For each of them either the startup moment (marked with @) or the time taken to start (marked with +) is shown.
root@hostname:~# systemd-analyze critical-chain user@0.service
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
`-systemd-logind.service @5.344s +625ms
`-basic.target @5.045s
`-sockets.target @5.033s
`-sshd.socket @4.916s +99ms
`-sysinit.target @4.803s
`-systemd-vconsole-setup.service @7.077s +191ms
`-systemd-journald.socket @1.474s
`--.mount @1.012s
`--.slice @1.012s
Similarly to the blame command, the output does not show simple units; moreover, parallelization is not really taken into account, and thus a unit startup time can be less (but also more) of the sum of the times taken by the previous units.
systemd-analyze plot
The systemd-analyze plot command writes to stdout a SVG graph depicting the entire boot sequence, with startup time and duration of every unit. It can be used to visualize sequentiality and parallelization of the various units, as well as the overall impact of each of them.
Command: systemd-analyze dot
The systemd-analyze dot command writes to stdout a graph in .dot format with all the dependencies among units. While very useful to understand a system, such graph can be intimidating on complex systems due to the high number of nodes and connections between them.
Once written to a file, the graph can be converted e.g. to a PNG using the following command:
dot -O -Tpng graph.dot
Trim unnecessary features
Being adopted in very different scenarios, from desktop PCs to servers to embedded devices, systemd includes a very high number of features, part of which might not be required, especially on an embedded system. Some boot time (and overall complexity!) can be saved disabling the features that are not required at build time: this leads to a smaller init system with fewer dependencies and components.
Most of the features can be disabled at build time using configure options; a bit of experimentation is probably required to obtain the exact configuration for a specific system, also considering that some features has loose dependency on others and have fallbacks with reduced features (e.g.: the lack of both polkit and logind can make the usage of non-root users a bit tricky).
Here is a non-exhaustive list of features that might not be needed on a embedded system:
| Feature | Disable with |
|---|---|
| hibernation support | -Dhibernate=false |
| systemd-repart | -Drepart=disabled |
| systemd-sysupdate | -Dsysupdate=disabled |
| systemd-portabled | -Dportabled=false |
| polkit support | -Dpolkit=disabled |
| systemd-importd | -Dimportd=disabled |
| systemd-journal-gatewayd | -Dmicrohttpd=disabled |
| FIDO2 support | -Dlibfido2=disabled |
| NSS support | -Dnss-systemd=false -Dnss-myhostname=false -Dnss-mymachines=disabled -Dnss-resolve=disabled |
| systemd-homed | -Dhomed=disabled |
| systemd-mountfsd | -Dmountfsd=false |
| systemd-userdbd | -Duserdb=false |
| UTMP support | -Dutmp=false |
| SysVinit compatibility | -Dsysvinit-path= -Dsysvrcnd-path= |
| rfkill support | -Drfkill=false |
| backlight persistence | -Dbacklight=false |
| systemd-sysusers | -Dsysusers=false |
Yocto HowTo
On Yocto, the systemd recipe exposes most of the tunable feature through the PACKAGECONFIG mechanism. While some of them are linked to the content of DISTRO_FEATURES, the features that get enabled by default are many: for this reason, some time should be taken to review all the enabled features and decide what can be removed. The best approach is to re-define the PACKAGECONFIG variable at distro level (i.e., inside the .conf file where the distro is defined) to include only the required features, e.g.:
PACKAGECONFIG:pn-systemd = "acl pam selinux binfmt gshadow logind networkd set-time-epoch"
It should be noted that to remove a feature means to recompile systemd, and then all the components that depend on it. This can become very costly in term of build time, especially when the dependent components are huge (e.g.: the Qt framework). For this reason, it is advisable to use a minimal image (e.g.: core-image-minimal) when doing this optimization.
Tuning behaviour with environment variables
Some configuration variables can affect systemd boot performances:
- SYSTEMD_DEFAULT_MOUNT_RATE_LIMIT_BURST: Mount commands in systemd are rate-limited but the default rate limit is quite low (5) so it's easy so exceed that and have systemd just sleeping during the bootup waiting for the limit to be matched.
Udev tuning
One of the more time-consuming steps during boot for a systemd/udev system is device cold plug. Udev will scan all devices (there could be a lot of them) and try to trigger the "add" uevent and process all the rules for every of them. A recent udev patch added the --include-parents flag to the trigger subcommand options, which could be used together with -s <subsystem> to filter by subsystem during initial cold plug. If only relevant subsystems are cold plugged in this way, it could shave off multiple seconds.
Here is an example udevadm command line:
* 'udevadm trigger --include-parents -s block -s tty -s drm -s input -s video'