Ftrace is the Linux kernel internal tracer that was included in the Linux kernel in 2.6.27. Although Ftrace is named after the function tracer it also includes many more functionalities. But the function tracer is the part of Ftrace that makes it unique as you can trace almost any function in the kernel and with dynamic Ftrace, it has no overhead when not enabled.
The interface for Ftrace resides in the debugfs file system in the tracing directory. Documentation for this can be found in the Linux kernel Documentation directory in trace/ftrace.txt.
Using the Ftrace debugfs interface can be awkward and time consuming. trace-cmd was created to interface with Ftrace using a binary tool which comes with full documentation in man pages.
Here's some examples of trace-cmd:
# trace-cmd record -e sched myprogram
The above will enable all the Ftrace tracepoints that are grouped under the sched system. You can find these tracepoints by looking at the debugfs system:
# mount -t debugfs nodev /sys/kernel/debug # ls /sys/kernel/debug/tracing/events/sched enable sched_process_fork sched_stat_sleep filter sched_process_free sched_stat_wait sched_kthread_stop sched_process_wait sched_switch sched_kthread_stop_ret sched_signal_send sched_wait_task sched_migrate_task sched_stat_iowait sched_wakeup sched_process_exit sched_stat_runtime sched_wakeup_new
trace-cmd allows you to see the possible events without needing to look at this directory as well.
# trace-cmd list -e | grep sched: sched:sched_kthread_stop sched:sched_kthread_stop_ret sched:sched_wait_task sched:sched_wakeup sched:sched_wakeup_new sched:sched_switch sched:sched_migrate_task sched:sched_process_free sched:sched_process_exit sched:sched_process_wait sched:sched_process_fork sched:sched_signal_send sched:sched_stat_wait sched:sched_stat_runtime sched:sched_stat_sleep sched:sched_stat_iowait
You can find trace-cmd in its git repository.
Also within that same repository is KernelShark, which is a graphical user interface to trace-cmd. trace-cmd is built with just "make" and KernelShark is created with "make gui". This allows building trace-cmd on your embedded device and keeping the build from needing the GTK libraries required by KernelShark.
Tracing a specific process with the Ftrace interface
(Adapted from email by Steven Rostedt) To trace just the kernel functions executed in the context of a particular function, set the pseudo-variable 'set-ftrace-pid', to the process id (pid) of the process.
If the process is not already running, you can use a wrapper shell script and the 'exec' command, to execute a command as a known pid.
#!/bin/sh echo $$ > /debug/tracing/set_ftrace_pid # can set other filtering here echo function > /debug/tracing/current_tracer exec $*
In this example, '$$' is the pid of the currently executing process (the shell script. This is set into the 'set_ftrace_pid' variable, then the 'function' tracer is enabled. Then this script exec's the command (specified by the first argument to the script).
Example usage (assuming script is called 'trace_command'):
Tracing a specific process with trace-cmd
# trace-cmd record -p function -F ls