Kernel Trace Systems

Here are some links to information about different kernel tracing systems:

General Purpose tracing systems
Some major Linux general-purpose tracing systems are:
 * ptrace - ability to trace syscall entry and exit, and signal delivery, to a process (also used for debugging a process)
 * see "man ptrace" and "man strace"
 * ftrace
 * tracer for kernel functions
 * can also be used for debugging or analyzing latencies and performance issues
 * in mainline since 2.6.27
 * See Measuring Function Duration with FTrace - a paper (under construction) by Tim Bird for Linux Symposium 2009
 * System Tap - System Tap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system
 * LTTng - Linux Trace Toolkit, next generation
 * LKST - Linux Kernel State Tracer

Special Purpose tracing systems
There are some other notable special-purpose kernel tracing systems:
 * KFT - Kernel Function Trace - traces functions to show function durations and call graphs
 * latency trace - RT-preempt tool for measuring interrupt and mutex latency
 * The latency tracer is embedded in the RT-preempt patch - see Realtime Preemption and RT-preempt Article
 * block tracer (blktrace) - allows you to see exactly what is going on in the block layer for a given queue
 * Introduction by Jens Axboe: Introduction
 * Execellent presentation: blktrace.pdf
 * Guide to using is at: blktrace guide
 * This appears to have been mainlined as of 2.6.17
 * Timeline utility (blktrace post-processing tool): blktrace timeline utility
 * delay accounting patches - collect statistics about the delays that are experienced by each task on the system
 * see delay accounting patches

Trace Infrastructure

 * KProbes - grew out of dprobes, with information at: dprobes
 * see an excellent tutorial at: kprobes
 * The mainline version of the KProbes supports x86,Alpha and PPC64 architectures. A MIPS implementation has been completed on the 2.6.16 kernel and tested on the Toshiba TX49 platform. Patch is available in the Patch Archive.


 * [would be nice to have some djprobe stuff here]

Sampling Systems
Note that profile systems (or "sampling systems") are slightly different, in that they involve sampling instead of event tracing. Some major ones for Linux are:
 * top - provides a dynamic real-time view of a running system, including processes
 * see "man top"
 * see also ksysguard, Gnome system Monitor
 * OProfile - system-wide profiler for Linux systems
 * see oprofile
 * also: oprofile at IBM
 * !BootChart - samples bootup and provides visualization of process startup and system utilization
 * see Boot Chart

Related facilities

 * in-kernel statistics infrastructure - proposal for a generic implementation of statistics facilities inside the kernel
 * see inkernel stats
 * perfmon2 - interfaces to hardware performance monitoring features of the CPU
 * see perfmon
 * inotify - inotify

Other Systems
Here are some systems I haven't classified yet:
 * Datastreams - a system for creating and monitoring tracepoints - see datastreams

Collaboration Efforts
Some trace system project leaders are trying to collaborate: see Tracing Collaboration Project