Kernel Trace Systems

Revision as of 16:31, 17 November 2009 by Tim Bird (Talk | contribs) (General Purpose tracing systems)

Jump to: navigation, search

Here are some links to information about different kernel tracing systems:

General Purpose tracing systems

Some major Linux general-purpose tracing systems are:

  • ptrace - ability to trace syscall entry and exit, and signal delivery, to a process (also used for debugging a process)
    • see "man ptrace" and "man strace"
  • ftrace
    • Ftrace Function Graph ARM - presentations and patches by Tim Bird to add function graph and duration tracing to ARM systems
      • The presentation has good introductory material on ftrace, as well as links to additional resources
    • tracer for kernel functions
    • can also be used for debugging or analyzing latencies and performance issues
    • in mainline since 2.6.27
    • See Measuring Function Duration with FTrace - outline of presentation by Tim Bird for Linux Symposium 2009
  • System Tap - System Tap is a system for building and executing tracing and sampling systems that can be applied to a running Linux system
  • LTTng - Linux Trace Toolkit, next generation
  • LKST - Linux Kernel State Tracer

Special Purpose tracing systems

There are some other notable special-purpose kernel tracing systems:

  • KFT - Kernel Function Trace - traces functions to show function durations and call graphs
  • latency trace - RT-preempt tool for measuring interrupt and mutex latency
  • block tracer (blktrace) - allows you to see exactly what is going on in the block layer for a given queue
  • delay accounting patches - collect statistics about the delays that are experienced by each task on the system

Trace Infrastructure

  • KProbes - grew out of dprobes, with information at: dprobes
    • see an excellent tutorial at: kprobes
    • The mainline version of the KProbes supports x86,Alpha and PPC64 architectures. A MIPS implementation has been completed on the 2.6.16 kernel and tested on the Toshiba TX49 platform. Patch is available in the Patch Archive.
  • [would be nice to have some djprobe stuff here]

Sampling Systems

Note that profile systems (or "sampling systems") are slightly different, in that they involve sampling instead of event tracing. Some major ones for Linux are:

  • top - provides a dynamic real-time view of a running system, including processes
  • OProfile - system-wide profiler for Linux systems
  •  !BootChart - samples bootup and provides visualization of process startup and system utilization

Related facilities

  • in-kernel statistics infrastructure - proposal for a generic implementation of statistics facilities inside the kernel
  • perfmon2 - interfaces to hardware performance monitoring features of the CPU
  • inotify - inotify

Other Systems

Here are some systems I haven't classified yet:

  • Datastreams - a system for creating and monitoring tracepoints - see datastreams

Collaboration Efforts

Some trace system project leaders are trying to collaborate: see Tracing Collaboration Project