Kernel Instrumentation

Here is a listing of some instrumentation systems for the kernel:

Printk Times
Produces printk's with extra time data on them. As of kernel 2.6.11 this is part of the mainline kernel enabled by CONFIG_PRINTK_TIME. Previous versions can add it via a very simple patch. It works for bootup time measurements, or other places where you can just jam in a printk or two.

See Printk Times

Boot Tracer
Starting from 2.6.28 the kernel has this new feature to optimize the boot time. It records the timings of the initcalls. Its aim is to be parsed by the scripts/bootgraph.pl tool to produce graphics about boot inefficiencies, giving a visual representation of the delays during initcalls. Users need to boot with the "initcall_debug" and "printk.time=1" parameters, and run "dmesg | perl scripts/bootgraph.pl > output.svg" to generate the final data.

Kernel Function Instrumentation (KFI)
A system which uses a compiler flag to instrument most of the functions in the kernel. Timing data is recorded at each function entry and exit. The data can be extracted and displayed later with a command-line program.

The kernel portion of this is available in the CELF tree now.

Grep for CONFIG_KFI.

See the page Kernel Function Instrumentation page for some preliminary notes.

FIXTHIS - need to isolate this as a patch.

Linux Trace Toolkit
See Linux Trace Toolkit

Kernel Tracer (in IKD patch)
This is part of a general kernel tools package, maintained by Andrea Arcangeli.

See http://www.kernel.org/pub/linux/kernel/people/andrea/ikd/README

The ktrace implementation is in the file kernel/debug/profiler.c It was originally written by Ingo Molnar, Richard Henderson and/or Andrea Arcangeli

It uses the compiler flag -pg to add profiling instrumentation to the kernel.

Function trace in KDB
Last year (Jan 2002) Jim Houston sent a patch to the kernel mailing list which provides support compiler-instrumented function calls.

See http://www.ussg.iu.edu/hypermail/linux/kernel/0201.3/0888.html

ftrace
Ftrace is a simple function tracer which initially came from the -rt patches but was mainlined in 2.6.27. Compiler profiling features are used to insert an instrumentation call (with gcc -pg option) that can be overwritten with a NOP sequence to ensure overhead is minimal with tracing disabled (this is enabled through CONFIG_DYNAMIC_FTRACE). There are a number of tracers in the kernel that use ftrace to trace high level events such as irq enabling/disabling preemption enabling/disabling, scheduler events and branch profiling.

The interface to access ftrace can be found in /debugfs/tracing, and is very extensively documented in Documentation/trace/ftrace.txt.

Steven Rostedt, ftrace's main developer, gave a conference on ftrace at the Ottawa Linux Symposium 2008. The video and slides.

SystemTap / Kprobes
SystemTap is a sophisticated kernel instrumentation tool that can be scripted with its own language to gather information about a running kernel. It uses the Kprobes infrastructure to implement it's tracing.