ELC 2025 Boot Time Presentations

Boot Time BOF

Speaker: Tim Bird, Sony
Site: https://ossna2025.sched.com/event/1zfle/boot-time-bof-tim-bird-sony
PDF: https://static.sched.com/hosted_files/ossna2025/6e/Boot-Time-BOF-Bird-ELC-2025-06.pdf
Video: https://youtu.be/qcotfSsrtnk

Technical Summary

• Focus: Reducing Linux boot time through better instrumentation, measurement, and kernel mechanisms.

Key Technical Topics

1. Instrumentation & Measurement

• Unified Boot Log (TI): Uses CPU cycle counters to measure firmware → kernel → user space time without timers.
• Cycle Counter Printk: Replace timestamp-based printk with cycle counters for early-boot accuracy.
• Bootstage Stash Driver: Reads and formats U-Boot timing data within the kernel.
• Better Probe Descriptions: Add function/file info to probe outputs for easier profiling.

2. Tools

• show_delta: Compares boot message timestamps.
• analyze_initcall_debug: Displays initcall timing.
• Red Hat tools for boot visualization.

3. Testing

• Boot-time regression and printk/initcall-based tests in development.
• Collabora’s test uses ftrace + bootconfig (requires CONFIG_FTRACE).

4. Kernel Patches & Mechanisms

• Deferred init calls: Delay non-critical initcalls to speed up boot.
• Deferred memory init: Initialize part of memory later (Qualcomm).
• Async probing: Drivers can probe in parallel, but unstable in many cases.
• Device dependencies: Defined via device tree “devlink” relationships.

5. Optimization Strategies

• Prioritize device initialization (e.g., defer USB or non-essential drivers).
• Kernel size reduction = direct boot time gains.
• “Boot cache” concept — avoid redundant driver probing by caching probe results.
• Similar to snapshot boot/unhibernate but without full memory restoration.

6. Data & Automation

• grab-boot-data.sh: Collects boot timing data for analysis and comparison.
• Boot Time Wiki stores logs, configs, and performance metrics.
• Goal: automate regression detection and document effective optimizations.

Early Boot and Late Attach of Remote Processors for Time-Critical Applications

Speaker: Vaishnav Achath, Beleswar Prasad Padhi, TI
Site: https://ossna2025.sched.com/event/1zfkC/early-boot-and-late-attach-of-remote-processors-for-time-critical-applications-beleswar-prasad-padhi-vaishnav-mohandas-achath-texas-instruments
PDF: https://static.sched.com/hosted_files/ossna2025/5a/oss_na_earlyboot_lateattach.pdf
Video: https://youtu.be/URw0l2n3fBk

Technical Summary

Problem

• Multi-core SoCs (e.g., TI AM64) include application, real-time, and DSP cores.
• In standard Linux boot flow, remote processors (R5/DSP) start late → delays early display/audio (needs <200 ms).
• Objective: Reduce latency by booting real-time cores earlier while keeping Linux control for IPC and complex tasks.

Solution: Early Boot + Late Attach Model

• Early Boot: Boot remote processor firmware from bootloader (U-Boot or ROM).
  • Configure clocks, reset lines, load firmware, map MMU tables, and start the core.
  • Only handle carveouts + device mappings (no IPC setup yet).
• Late Attach: When Linux kernel boots:
  • Skip resetting cores already running.
  • Reattach to remote processors to set up IPC (virtio/rpmsg).
  • Restore resource table, register trace buffers, and reinitialize communication.

Key Mechanisms

• Resource Table: Firmware advertises required carveouts, virtio devices, trace buffers, and mappings.
• VirtIO + RPMsg: Framework for zero-copy inter-processor communication via shared DDR buffers and mailboxes.
• Late Attach Implementation:
  • Remoteproc framework extended to detect pre-booted cores and use existing resource tables.
  • MMU drivers retrieve page tables from bootloader (TTB registers).
  • Timer drivers detect running counters and avoid reconfiguration (prevent crashes).

Challenges

1. Core State Detection:

• TI uses LPSC registers to check if cores are powered on.
• Xilinx and NXP use memory or DT-based indicators.

2. Resource Table Retrieval:

• Address stored at base of carveout; passed from bootloader to kernel.

3. Suspend/Resume:

• Depends on platform host-ID logic (e.g., TI’s device manager prevents mismatched shutdown).

Debugging Lessons

• Avoid overlapping reserved carveouts with kernel decompression area (e.g., 16 MB offset in ARM32).
• Preserve MMU tables — prevent dma_alloc_coherent() from clearing reserved memory (added non-zeroing flag).
• Check for platform-specific code resetting peripherals before drivers probe.
• Validate runtime power/clock states (don’t rely only on static DT).

Result

• Remote cores booted from U-Boot continue running; Linux attaches later for IPC setup.
• Enables early camera/display/audio (<200 ms) and complex runtime (3D surround, image stitching).
• Fully compatible with TI SoCs and generic remoteproc + virtio frameworks.

Early Ethernet With Linux

Speaker: Keerthy Jagadeesh, TI
Site: https://ossna2025.sched.com/event/1zfjQ/early-ethernet-with-linux-keerthy-jagadeesh-texas-instruments
PDF: https://static.sched.com/hosted_files/ossna2025/dd/Early%20ethernet%20with%20Linux-final.pdf
Video: https://youtu.be/id-EB9j8Wig

Technical Summary

Goal

Enable Ethernet initialization within ~2 seconds on automotive gateway SoCs using Linux — critical for zonal/SDV architectures where other ECUs depend on network availability.

Context

• Transition from domain-based to zone-based automotive architectures → Ethernet becomes the backbone interconnect.
• Central gateway SoC handles all ECU communication; it must bring up Ethernet early for OTA updates, inter-ECU communication, and telemetry.

Hardware Architecture

• SoC: TI DRA821 (dual Cortex-A72, multiple R5F, M3 cores, integrated CPSW Ethernet switch).
• Linux on A72: runs gateway stack and user-space networking.
• MCU R5F: initial boot core (runs ATOS, device management, clock/power control).
• Main R5F: used later for auxiliary tasks (Ethernet firmware, offload, etc.).
• M3: manages secure boot and runtime authentication.

Software Architecture & Boot Flow

1. ROM → SPL (Secondary Bootloader):

• Initialize boot media (OSPI/eMMC).
• Load ATF + Linux directly (Falcon boot mode).
• Skip full U-Boot for faster boot.

2. Bootloader Optimizations:

• Use OSPI for fastest startup (~10–20 ms).
• Remove unused configs and disable console prints.
• Trim kernel (defconfig → ~10–12 MB).
• Use minimal rootfs (BusyBox hybrid).

3. Linux Boot Optimizations:

• Disable unused device tree nodes.
• Disable verbose kernel logs.
• Use eMMC/UFS for root filesystem.
• Optimize systemd-networkd startup and VLAN setup.

Ethernet Driver Optimizations

• Disable Auto-Negotiation: fixed PHY setup saves ~1–2 s.
• Move ndo_open() (device bring-up) from user-space init to driver probe(), enabling earlier MAC/PHY config (~2 s saved).
• Increase PHY polling frequency (from 1 s → 100 ms).
• Enable CPSW internal switch earlier → other ECUs can communicate before Linux userspace loads.

Parallel Boot Using R5 Firmware

• A dedicated R5 Ethernet firmware initializes CPSW + PHY independently before Linux boots.
• Provides a virtual Ethernet interface to Linux (late attach).
• Reduces internal switch initialization to ~500 ms; Linux Ethernet ready by ~1.8 s.
• Firmware includes lightweight TCP/IP (LWIP) stack for early communication if needed.

Results

Phase	Action	Time
Bootloader (ROM → SPL → ATF)	System init + security	~300 ms
R5 Ethernet Firmware Init	CPSW + PHY configured	~500 ms
Linux Attach + Driver Probe	MAC up + IP config	~1.2 s
Total (Full Linux Ethernet Ready)		~1.8 s

Key Takeaways

• Use OSPI boot + Falcon mode for minimal boot overhead.
• Trim device tree and kernel for faster Linux init.
• Disable PHY autoneg and bring up link at driver probe stage.
• Use auxiliary R5 core for pre-Linux Ethernet bring-up and parallelization.
• Virtual Ethernet link allows Linux to attach later without resetting CPSW.

Improving Linux Boot Time for Automotive Use Cases

Speaker: Brian Masney & Eric Chanudet, Red Hat
Site: https://ossna2025.sched.com/event/1zfin/improving-linux-boot-time-for-automotive-use-cases-brian-masney-eric-chanudet-red-hat
PDF: https://static.sched.com/hosted_files/ossna2025/3d/2025-ELC-Masney-Chanudet-Boot-Time.pdf
Video: Improving Linux Boot Time for Automotive Use Cases - Brian Masney & Eric Chanudet, Red Hat

Technical Summary

Scope

• Focus: kernel entry → device initialization, not firmware/bootloader.
• Goal: Reduce kernel + user-space boot to under ~2 seconds for automotive SoCs.
• Measurement: Using only kernel logs, trace events, kprobes, systemd-analyze, etc.—no hardware probes.

Kernel-Level Optimizations

1. Kernel Configuration

• Remove unused subsystems and debugging options to reduce initcalls.
• Disable console logging (printk) and serial outputs → saves tens of ms.
• Disable crypto/FIPS self-tests and other QA-only validations.
• Trim defconfig → smaller image → faster load from storage.

2. Linear Mapping & Memory Init

• ARM64 setup_arch() spent ~350 ms creating the linear map on 32 GB platforms.
• Optimization: remove redundant memory barriers/TLB invalidations → reduced to 25 ms.

3. Deferred struct page Init

• CONFIG_DEFERRED_STRUCT_PAGE_INIT moves page struct setup later when SMP is ready.
• Added multi-core awareness for ARM → improved parallelization (~20–30 ms saved).

4. RCU & Synchronization

• RCU barriers block parallel init under PREEMPT_RT.
• Switched to expedited RCU during boot, then reverted post-init → ~100 ms saved.
• Found redundant RCU sync loops costing ~100 ms — replaced with workqueue completion.

5. Driver-Specific Optimizations

• UFS controller: Skip speed renegotiation if firmware already configured → ~50 ms saved.
• Ethernet PHY: Remove redundant GPIO resets and polling delays → significant link-up improvement.
• General rule: avoid reinitialization if firmware left hardware ready.

6. Kernel Compression

• Use uncompressed kernel (on fast flash) → avoids decompression overhead.
• Use LZ4 for initramfs and ZSTD for modules (faster decompression).

Userland (Init & Systemd) Optimizations

1. Minimal Initramfs

• Build only storage/network drivers into kernel → no module load delays.
• Avoid long-lived daemons in initramfs — only mount rootfs and switch.
• Remove journald and udev from initramfs (too slow).
• Custom lightweight init (mount-sysroot) replaces full systemd init.

2. Systemd & SELinux

• Minimize SELinux policy (remove unused domains) → ~50% faster policy load.
• Pre-generate systemd unit files for known hardware targets.
• Optimize udev:
  • Trim unused rules.
  • Avoid external exec calls in rules.
  • Filter coldplug by subsystem.
  • Increase mount concurrency (DefaultLimitBurst).

3. Initramfs vs Direct Rootfs

• On many boards, having a minimal initramfs is faster than direct boot:
  • Allows parallel init while UFS/eMMC initializes.
  • Direct rootfs boot stalls kernel waiting for storage readiness.

Measurement Tools

• systemd-analyze plot → visualize init sequence and timing.
• systemd-bootchart → shows CPU/IO utilization and process startup.
• boot-time-analysis (Red Hat) and analyze-init-debug → kernel timestamp tracing.

Results

Stage	Optimization	Result
Kernel + Drivers	Deferred init, RCU, driver fixes	< 400 ms
Initramfs + RootFS Switch	Custom minimal init	~200–300 ms
Full Userland (Systemd Ready)	Optimized SELinux + udev	~1.8–2.0 s total

Key Takeaways

• Disable logs, QA checks, and redundant probing.
• Parallelize init (deferred page + multi-core).
• Use RCU expedited early in boot.
• Minimize kernel & initramfs size.
• Trim systemd and SELinux for automotive or embedded targets.
• Avoid reinitializing hardware already configured by firmware.

Boot Time Regression Testing in Linux

Speaker: Tim Bird, Sony
Site: https://ossna2025.sched.com/event/1zfih/regression-testing-boot-time-performance-in-the-linux-kernel-tim-bird-sony
PDF: https://static.sched.com/hosted_files/ossna2025/b7/Regression-testing-Boot-Time-Performance%20in%20the%20Linux-Kernel-Bird-ELC-2025-06.pdf
Video: https://youtu.be/jgrBv6EBb6o

Technical Summary – Linux Boot Time Regression Testing

Goal

Develop an automated upstream test framework to detect Linux kernel boot time regressions using measurable, reproducible metrics instead of manual inspection.

Motivation

• There are many boot analysis tools (bootchart, bootgraph, FTrace, printk timestamps) — but no automated test that reports pass/fail results for boot-time regressions.
• Boot time depends on:
  • Hardware variation (CPU count, memory, I/O, bus topology).
  • Kernel config and image size.
  • Use case (e.g. camera init for automotive, Ethernet init for networking).
• Therefore, one shared test must be configurable and metadata-aware.

Challenges

1. No common timing points: Printk messages differ across architectures/subsystems. 2. Blind spots: 60–150 messages timestamped as “0” before clock init (≈100–400 ms unaccounted). 3. Deferred probes: Many device init delays happen outside measurable initcall functions. 4. No unified “boot region” markers across subsystems. 5. Automation requires kernel cmdline mods (initcall_debug) — varies by bootloader (GRUB, U-Boot, etc.).

Proposed Framework

1. Data Collection – grab_boot_data.sh

• Simple POSIX shell script for any embedded system (requires only BusyBox).
• Collects:
  • Kernel boot log (dmesg, initcall_debug).
  • Metadata: kernel version, config, CPU/mem info, storage, etc.
• Stores locally and optionally uploads to a Boot Data Wiki for aggregation.

2. Analysis – boot_time_regression_test

• Compares current boot log with a reference value file.
• Detects regressions using thresholds (default: +5%, >30 µs).
• Outputs in KTAP format for automated reporting.

3. Reference Value Files

• Store known-good timing data and metadata.
• Used for pass/fail comparison.
• Can be local or fetched automatically from Boot Data Wiki based on metadata matching.
• Matching algorithm uses scoring (same board, config, lab, kernel version, etc.).

Regression Detection Example

• Compared Ubuntu 20.04 (kernel 5.15) → 24.04 (kernel 6.8):
  • Found 10+ initcalls >10 ms slower (e.g. input LED init grew from 0.5 ms → 30 ms).
  • Example: jitterentropy_init rose 30 → 50 ms (kernel 6.8 → 6.12).

Future Enhancements

• Use cycle counters (hardware stable timers) for sub-ms precision before clock init.
• Support FTrace-based regions once early tracing improves.
• Add deferred probe instrumentation (RFC patches in progress).
• Expand to include bootloader and systemd (analyze blame) phases.
• Build an automated boot tuning tool using accumulated regression data.

Key Components

Tool	Function
grab_boot_data.sh	Collect kernel logs + metadata
boot_time_regression_test	Compare to reference and flag regressions
find_matching_ref_value	Match test data to best reference
boot_data_parser	Parse + visualize results
Boot Data Wiki	Central database of boot metrics, reports, and reference values

Key Takeaways

• Automated regression testing is possible using initcall + printk data.
• Requires robust metadata and reference matching due to hardware/config diversity.
• Cycle counters and early tracing can close measurement gaps.
• Shared boot data and reference sets enable cross-vendor boot-time benchmarking.

U-Boot’s new standard boot and what’s next

Speaker: Simon Glass, Canonical
Site: https://ossna2025.sched.com/event/1zff4/u-boots-new-standard-boot-and-whats-next-simon-glass-canonical
PDF: https://static.sched.com/hosted_files/ossna2025/99/U-Boot%20Standard%20Boot%20and%20What%27s%20Next.pdf
Video: U-Boot's New Standard Boot and What's Next - Simon Glass, Canonical

Technical Summary

Current State

• U-Boot acts as a universal bootloader with:
  • Broad board and QEMU support.
  • EFI loader compatibility (runs EFI apps on ARM/x86).
  • Integration with Device Tree (DT) and SystemReady initiative.
• CI and testing have improved, but firmware complexity continues to grow — multiple components (TF-A, OP-TEE, vendor blobs) must be bundled.

Problems

1. Messy environment – legacy scripting (DRO scripts) and inconsistent boot logic. 2. EFI complexity – large, slow code path with redundant logic. 3. Slow boot due to device initialization (pinmux, probing pre-relocation). 4. Device Tree security mismatch – DT must match Linux kernel yet resides in firmware. 5. Firmware image sprawl – multiple binaries stitched manually. 6. Limited automation – many boards can break from generic code.

Standard Boot Framework

A simplified and unified U-Boot boot model:

• Removes dependency on legacy scripts (bootcmd, DRO).
• Defines bootflow as a structured boot pipeline:
  • bootdev → where OS is located (storage, network, USB).
  • bootmeth → how to find/load the OS (e.g. extlinux, EFI, Android).
  • bootflow struct → metadata of loaded OS (DT, initrd, command line).
• Single API: bootflow get → enumerate or boot OS.

Features

• Auto boot discovery: scans devices and picks valid OS images.
• Prioritizes fast devices first (e.g. eMMC over USB).
• bootflow command: inspect available OSes, view boot args, boot selected one.
• expo UI system: text or graphical menu for OS selection (replaces ugly menus).
• No command-line needed – boots programmatically.
• Tests: 5,000+ lines of CI tests for OS scanning, EFI service checks, and memory allocation validation.

Performance Work

• Demonstrated U-Boot-to-Linux handoff in ~82 ms on QEMU/KVM.
• Eliminated CLI delay for production images.
• Cloud boot optimization experiments targeting sub-100 ms boot.

Tools & Frameworks

1. Binman

• Declarative tool for building complex firmware images.
• YAML-style description of components (TF-A, SPL, OP-TEE, U-Boot, DT, signatures).
• Handles blob imports, tool fetching, and image layout automatically.
• Simplifies vendor firmware assembly and reproducibility.

2. LabGrid Integration

• Automates hardware testing and CI integration.
• Supports U-Boot test execution on real boards via GitLab runners.

Future & Vision

Goals

• Fully open-source boot chain from reset to Linux entry.
• Eliminate redundant EFI layers (shim, grub, systemd-stub).
• Move toward Flat Image Tree (FIT) or Universal Payload (UPL) models:
  • Combine kernel, DTs, and initrd in one self-describing image.
  • Avoid FAT filesystem & EFI callbacks.
• Predeclare boot dependencies (entropy, RNG, relocation) — no runtime callbacks.
• Require ext4 and modern storage support as baseline.

Ideal Model

• Open firmware declares supported features → OS negotiates requirements pre-boot.
• No opaque EFI callbacks or external binaries.
• Secure boot chain + simple interface = Linux-ready state immediately after firmware.

Key Takeaways

• Standard Boot simplifies and unifies boot process across boards.
• Binman provides reproducible, declarative firmware image construction.
• Cycle-time optimization: U-Boot can reach Linux in <100 ms.
• Goal: Open, deterministic, minimal boot firmware without EFI dependency.