Ftrace

Ftrace is the Linux kernel internal tracer that was included in the Linux kernel in 2.6.27. Although Ftrace is named after the function tracer it also includes many more functionalities. But the function tracer is the part of Ftrace that makes it unique as you can trace almost any function in the kernel and with dynamic Ftrace, it has no overhead when not enabled.

The interface for Ftrace resides in the debugfs file system in the tracing directory. Documentation for this can be found in the Linux kernel Documentation directory in trace/ftrace.txt.

trace-cmd
Using the Ftrace debugfs interface can be awkward and time consuming. trace-cmd was created to interface with Ftrace using a binary tool which comes with full documentation in man pages.

Here's some examples of trace-cmd:


 * 1) trace-cmd record -e sched myprogram

The above will enable all the Ftrace tracepoints that are grouped under the sched system. You can find these tracepoints by looking at the debugfs system:

enable                 sched_process_fork  sched_stat_sleep filter                 sched_process_free  sched_stat_wait sched_kthread_stop     sched_process_wait  sched_switch sched_kthread_stop_ret sched_signal_send   sched_wait_task sched_migrate_task     sched_stat_iowait   sched_wakeup sched_process_exit     sched_stat_runtime  sched_wakeup_new
 * 1) mount -t debugfs nodev /sys/kernel/debug
 * 2) ls /sys/kernel/debug/tracing/events/sched

trace-cmd allows you to see the possible events without needing to look at this directory as well.

sched:sched_kthread_stop sched:sched_kthread_stop_ret sched:sched_wait_task sched:sched_wakeup sched:sched_wakeup_new sched:sched_switch sched:sched_migrate_task sched:sched_process_free sched:sched_process_exit sched:sched_process_wait sched:sched_process_fork sched:sched_signal_send sched:sched_stat_wait sched:sched_stat_runtime sched:sched_stat_sleep sched:sched_stat_iowait
 * 1) trace-cmd list -e | grep sched:

You can find trace-cmd in its git repository.

Also within that same repository is KernelShark, which is a graphical user interface to trace-cmd. trace-cmd is built with just "make" and KernelShark is created with "make gui". This allows building trace-cmd on your embedded device and keeping the build from needing the GTK libraries required by KernelShark.

Tracing a specific process with the Ftrace interface
(Adapted from email by Steven Rostedt) To trace just the kernel functions executed in the context of a particular function, set the pseudo-variable 'set-ftrace-pid', to the process id (pid) of the process.

If the process is not already running, you can use a wrapper shell script and the 'exec' command, to execute a command as a known pid.

Like so: echo $$ > /debug/tracing/set_ftrace_pid echo function > /debug/tracing/current_tracer exec $*
 * 1) !/bin/sh
 * 1) can set other filtering here

In this example, '$$' is the pid of the currently executing process (the shell script. This is set into the 'set_ftrace_pid' variable, then the 'function' tracer is enabled.  Then this script exec's the command (specified by the first argument to the script).

Example usage (assuming script is called 'trace_command'): trace_command ls

Tracing a specific process with trace-cmd

 * 1) trace-cmd record -p function -F ls