Boot Time

Introduction
Boot Time includes topics such as measurement, analysis, human factors, initialization techniques, and reduction techniques. The time that a product takes to boot directly impacts the first perception an end user has of the product. Regardless of how attractive or well designed a consumer electronic device is, the time required to move the device from off to an interactive, usable state is critical to obtaining a positive end user experience. Turning on a device is Use Case #1.

Booting up a device involves numerous steps and sequences of events. In order to use consistent terminology, the Bootup Time Working of the CE Linux Forum came up with a list of terms and their widely accepted definitions for this functionality area. See the following page for these terms:
 * Boot-up Time Definition Of Terms

Technology/Project Pages
The following are individual pages with information about various technologies relevant to improving Boot Time for Linux. Some of these describe local patches available on this site. Others point to projects or patches maintained elsewhere.

Measuring Boot-up Time

 * Printk Times - simple system for showing timing information for each printk.
 * Kernel Function Trace - system for reporting function timings in the kernel.
 * Linux Trace Toolkit - system for reporting timing data for certain kernel and process events.
 * Oprofile - system-wide profiler for Linux.
 * Bootchart - a tool for performance analysis and visualization of the Linux boot process. Resource utilization and process information are collected during the user-space portion of the boot process and are later rendered in a PNG, SVG or EPS encoded chart.
 * Bootprobe - a set of System Tap scripts for analyzing system bootup.
 * and, let us not forget: "cat /proc/uptime"
 * grabserial - a nice utility from Tim Riker to log and timestamp console output
 * process trace - a simple patch from Tim Riker to log exec, fork and exit system calls.

[ FIXTHIS - should mention (and publish) sstream here ? ]

[ FIXTHIS - should also mention initcall_debug (kernel command line option) here - it shows driver init times for many drivers]


 * See also: Kernel Instrumentation which lists some known kernel instrumentation tools. These are of interest for measuring kernel startup time.

Bootloader speedups

 * Kernel XIP - Allow kernel to be executed in-place in ROM or FLASH.
 * DMA Copy Of Kernel On Startup - Copy kernel from Flash to RAM using DMA
 * Uncompressed kernel - An uncompressed kernel might boot faster

Kernel speedups

 * Disable Console - Avoid overhead of console output during system startup.
 * Disable bug and printk - Avoid the overhead of bug and printk. Disadvantage is that you loose a lot of info.
 * RTC No Sync - Avoid delay to synchronize system time with RTC clock edge on startup.
 * Short IDE Delays - Reduce duration of IDE startup delays (this is effective but possibly dangerous).
 * Hardcode kernel module info - Reduce the overhead of loading a module, by hardcoding some information used for loading the relocation information
 * IDE No Probe - Force kernel to observe the ide=noprobe option.
 * Preset LPJ - Allow the use of a preset loops_per_jiffy value.
 * Threaded Device Probing - Allow drivers to probe devices in parallel.
 * Reordering of driver initialization - Allow driver bus probing to start as soon as possible.
 * NAND ECC improvement - The pre 2.6.28 nand_ecc.c has room for improvement. You can find an improved version in the mtd git at http://git.infradead.org/mtd-2.6.git?a=blob_plain;f=drivers/mtd/nand/nand_ecc.c;hb=HEAD. Documentation for this is in http://git.infradead.org/mtd-2.6.git?a=blob_plain;f=Documentation/mtd/nand_ecc.txt;hb=HEAD. This is only interesting if your system uses software ECC correction.
 * Check what kernel memory allocator you use. Slob or slub might be better than slab (which is the default in older kernels)
 * If your system does not need it, you can remove SYSFS and even PROCFS from the kernel. In one test removing sysfs saved 20 ms.
 * Carefully investigate all kernel configuration options on whether they are applicable or not. Even if you select an option that is not used in the end, it contributes to the kernel size and therefore to the kernel load time (assuming you are not doing kernel XIP). Often this will require some trial and measure! E.g. selecting CONFIG_CC_OPTIMIZE_FOR_SIZE (found under general setup) gave in one case a boot improvement of 20 ms. Not dramamtic, but when reducing boot time every penny counts!
 * Moving to a different compiler version might lead to shorter and/or faster code. Most often newer compilers produce better code. You might also want to play with compiler options to see what works best.

File System issues
Different file systems have different initialization (mounting) times, for the same data sets. This is a function of whether meta-data must be read from storage into RAM or not, and what algorithms are used during the mount procedure.


 * Filesystem Information - has information about boot-up times of various file systems
 * File Systems - has information on various file systems that are interesting for embedded systems. Also includes some improvement suggestions.
 * Avoid Initramfs - explains on why intramfs should be avoided if you want to minimize boot time
 * Split partitions. If mounting a file system takes long, you can consider splitting that filesystem in two parts, one with the info that is needed during or immediately after boot, and one which can be mounted later on.

User-space and application speedups
. This will allow you to create an optimised library. As unneeded functions are removed this should lead to a performance gain. Normally there will be library pages which contain unused code (adjacent to code that is used). After optimizing the library this does not occur any more, so less pages are needed and hence less page loads, so some time can be saved.
 * Optimize RC Scripts - Reduce overhead of running RC scripts
 * Parallel RC Scripts - Run RC scripts in parallel instead of sequentially
 * Application XIP - Allow programs and libraries to be executed in-place in ROM or FLASH
 * Pre Linking - Avoid cost of runtime linking on first program load
 * Statically link applications. This avoids the costs of runtime linking. Useful if you have only a few applications. In that case it could also reduce the size of your image as no dynamic libraries are needed
 * GNU_HASH: ~ 50% speed improvement in dynamic linking
 * See http://sourceware.org/ml/binutils/2006-06/msg00418.html
 * Application Init Optimizations - Improvements in program load and init time via:
 * use of mmap vs. read
 * control over page mapping characteristics.
 * Include modules in kernel image - Avoid extra overhead of module loading by adding the modules to the kernel image
 * Avoid udev. Udev takes quite some time to populate the /dev directory. In an embedded system it is often known what devices are present and in any case you know what drivers are available, so you know what device entries might be needed in /dev. These should be created statically, not dynamically. mknod is your friend, udev is your enemy.
 * Moving to a different compiler version might lead to shorter and/or faster code. Most often newer compilers produce better code. You might also want to play with compiler options to see what works best.
 * If possible move from glibc to uclibc. This leads to smaller executables and hence to faster load times.
 * library optimiser tool: http://libraryopt.sourceforge.net/
 * Function reordering: http://www.celinux.org/elc08_presentations/DDLink%20FunctionReorder%2008%2004.pdf

This is a technique to rearrange the functions within an executable so they appear in the order they are needed. This improves the load time of the application as all initialisation code is grouped into a set of pages, instead of being scattered over a number of pages.

Suspend related improvements
Another approach to improve boot time is to use a suspend related mechanism. Two approaches are known. Issue with this approach is that flash write is much slower than flash read, so the actual creation of the hibernate image might take quite a while. This is similar to hibernate and resume, but the hibernate file is retained and used upon every boot. Disadvantage is that no writable partitions should be mounted at the time of making the snapshot. Otherwise inconsistencies will occur if a partition is modified, while applications in the hibernate file might have information in the snapshot related to the unmodified partition.
 * Using the standard hibernate/resume approach. This is what has been demonstrated by Chan Ju, Park, from Samsung. See sheet 23 and onwards from this [[Media:LinuxBootupTimeReduction4DSC.ppt|PPT]] and section 2.7 of this paper.
 * Implementing snapshot boot. This is done by Hiroki Kaminaga from Sony and is described at snapshot boot for ARM and http://elinux.org/upload/3/37/Snapshot-boot-final.pdf

Uninvestigated speedups
This section is a holding pen for ideas for improvement that are not implemented yet but that could result in a boot time gain. Please leave a note here if you are working on one of these items to avoid duplicate work.

Unfortunately my knowledge of the internals in this section is not yet good enough to do a trial implementation. Caveats:
 * Prepopulated buffer cache - As initramfs performs an additional copy of the data the idea is to have a prepopulated buffer cache. A simplistic scenario would allow dumping the buffer cache when the booting is completed and the user applications have initialised. This data then could be used in a subsequent boot to initialize the buffer cache (of course without copying). A possible approach would be to have those data to reside into the kernel image and use them directly. Alternately they could be loaded separately.
 * is it possible to have the buffer cache split into two different parts, one which is statically allocated, one which is dynamically allocated?
 * the pages in the prepopulated buffer cache probably cannot be discarded, so they should be pinned
 * apart from the buffer cache data itself also some other variables might need restoring
 * a similar approach could also be used for the cached file data.
 * Dedicated fs - currently a lot of abstraction is done in fs to make a nice abstraction allowing easy addition of new filesystems and creating a unified view of those filesystem. While this is pretty neat, the abstraction layers also introduce some overhead. A solution could be to create a dedicated fs system, which supports only one (or maybe 2) filesystems, and eliminates the abstraction overhead. This will give some benefit, but the chance of getting this into the mainline is zero.

Articles and Presentations

 * "Tools and Techniques for Reducing Bootup Time" ([[Media:Tools-and-technique-for-reducing-bootup-time.ppt|PPT]] | [[Media:Tools-and-technique-for-reducing-bootup-time.odp|ODP]] | [[Media:Tools-and-technique-for-reducing-bootup-time.pdf|PDF]])
 * Tim Bird has presented at ELC Europe, on November 7, 2008, his latest collection of tips and tricks for reducing bootup time
 * Tims Fastboot Tools has online materials in support of this presentation
 * Optimizing Linker Load Times
 * (introducing various kinds of bootuptime reduction, prelinking, etc.)
 * Benchmarking boot latency on x86
 * By Gilead Ben-Yossef, July 2008
 * A tutorial on using TSC register and the kernel PRINTK_TIMES feature to measure x86 system boot time, including BIOS, bootloader, kernel and time to first user program.
 * Fast Booting of Embedded Linux
 * By HoJoon Park, Electrons and Telecommunications Research Institute (ETRI), Korea, Presented at the CELF 3rd Korean Technical Jamboree, July 2008
 * Explains several different reduction techniques used for different phases of bootup time
 * Tim Bird's (Sony) survey of boot-up time reduction techniques:
 * Methods to Improve Boot-up Time in Linux - Paper prepared for 2004 Ottawa Linux Symposium
 * - December 2003 Presentation describing some existing boot-up time reduction techniques and strategies.

Case Studies

 * Samsung proof-of-acceptability study for digital still camera: see [[Media:LinuxBootupTimeReduction4DSC.ppt|Boot Up Time Reduction PPT]] and the paper describing this.

Splash Screen projects
user-space code. embedded Linux devices supporting a 16bpp or 32bpp framebuffer.
 * Splashy - Technology to put up a spalsh screen early in the boot sequence. This is
 * This seems to be the most current splash screen technology, for major distributions. A framebuffer driver for the kernel is required.
 * Gentoo Splashscreen - newer technology to put a splash screen early in the boot sequence
 * See the HOWTO at: HOWTO FBSplash
 * PSplash - PSplash is a userspace graphical boot splash screen for mainly
 * bootsplash.org - put up a splash screen early in boot sequence
 * This project requires kernel patches
 * This project is now abandoned, and work is being done on Splashy.

Kexec

 * Kexec is a system which allows a system to be rebooted without going through BIOS. That is, a Linux kernel can directly boot into another Linux kernel, without going through firmware. See the white paper at: kexec.pdf
 * 2004 Kernel Summit presentation: fastboot.pdf
 * here's another Kexec white paper:Reboot Fast

Others

 * FSMLabs Fastboot - press release by FSMLabs about fast booting of their product. Is any of this published?


 * snapshot boot - a technology uses software resume to boot up the system quickly.

Apparently obsolete or abandoned material

 * [[Image:alert.gif]] in progress - Boot-up Time Reduction Howto - this is a project to catalog existing boot-up time reduction techniques.
 * Was originally intended to be the authoritative source for bootup time reduction information.
 * No one maintains it any more (as of Aug, 2008)
 * [[Image:alert.gif]]no content yet - Boot-up Time Delay Taxonomy - list of delays categorized by boot phase, type and magnitude
 * Was to be a survey of common bootup delays found in embedded devices.
 * Was never really written.