Linux Trace Toolkit - Revision history

Wmat: /* LKML Reaction to tracing */

2011-06-22T20:25:13Z

‎LKML Reaction to tracing

Tim Bird: change to tracing and kernel categories

2009-05-13T00:39:07Z

change to tracing and kernel categories

Tim Bird: Reverted edits by Tim Bird (Talk); changed back to last version by FransMeulenbroeks

2008-11-25T00:09:38Z

Reverted edits by Tim Bird (Talk); changed back to last version by FransMeulenbroeks

Tim Bird: Reverted edits by FransMeulenbroeks (Talk); changed back to last version by Tim Bird

2008-11-25T00:07:00Z

Reverted edits by FransMeulenbroeks (Talk); changed back to last version by Tim Bird

FransMeulenbroeks: /* LKML Reaction to tracing */

2008-11-24T20:33:07Z

‎LKML Reaction to tracing

Tim Bird: add download for LTT original on 2.6.11

2008-09-18T23:16:33Z

add download for LTT original on 2.6.11

Tim Bird: /* References */ add link to ols 2006 paper

2007-10-31T16:11:21Z

‎References: add link to ols 2006 paper

Wmat: fixing line returns

2006-12-05T21:21:46Z

fixing line returns

Wmat: fixing line returns

2006-12-05T21:20:37Z

fixing line returns

Wmat: Legacy Content Conversion

2006-12-05T21:19:31Z

Legacy Content Conversion

New page

@) '''''Preliminary Draft''''' @) under construction

== Introduction ==
The Linux Trace Toolkit is used to examine the flow of execution (between processes, kernel threads, and interrupts) in a Linux system. This is useful for analyzing where delays occur in the system, and to see how processes interact (especially with regard to scheduling, interrupts, synchronization primitives, etc.)

[[Image:Alert.gif]] * Note that as of 2.6.14, LTT is being replaced with LTTng. See [http://ltt.polymtl.ca/ LTTng]

See also the announcement here: [http://lwn.net/Articles/170198/ Patch: Linux Trace Toolkit Viewer/Next Generation announcement (LTTV/LTTng)]

=== Rationale ===
Tracing is useful for analyzing stuff.

== Downloads ==
=== LTTng ===
See the QUICKSTART guide at: [http://ltt.polymtl.ca/svn/ltt/branches/poly/QUICKSTART Quickstart]

=== Sample i386 trace for LTTng ===
Here is a zip file containing a sample trace on an i386 machine: i386-ltt-trace.zip

=== 0.9.6 Release (Nov 21, 2004) ===
I found it a bit difficult to get the appropriate set of patches for LTT. After a bit of frustration, I built my own release for Nov. 23, 2004. The release is 0.9.6 (no 'pre') and it works with the Linux kernel 2.6.9. The tar file for the release is in the PatchArchive.

=== Older stuff ===

The latest stable release is 0.9.5a, but this release is over 2 years old.

Here is the patch set I finally used, for kernel 2.6.8.1:
*[http://www.listserv.shafik.org/pipermail/ltt-dev/2004-August/000638.html message with ltt-2.6.8.1-relayfs.patch.bz2 by Mathieu Desnoyers]
*[http://www.listserv.shafik.org/pipermail/ltt-dev/2004-August/000637.html message with ltt-2.6.8.1-ltt.patch.bz2 by Mathieu Desnoyers]

=== getting attachments from ltt-dev archive ===
I had to use 'munpack' to extract the compressed patches from the mail messages in the archive. 'munpack' is part of the package 'mpack', which is available at: [ftp://ftp.andrew.cmu.edu/pub/mpack/ munpack]

=== Patch ===
*Patches for Linux kernel version 2.6.8.1, and !TraceToolkit tarfile are here:
**ltt-2.6.8.1.tar.gz
**TraceToolkit-0.9.6pre3-plus.tar.gz

These are bundles with multiple sub-patches and sub-tars. To install them, do the following:
*untar kernel patches:
**<code>tar -xzvf ltt-2.6.8.1.tar.gz</code>
*apply patches:
**<code>./tpm -t linux-2.6.8.1.tar.bz2 -f ltt-2.6.8.1.pl -o linux-2.6.8.1-ltt</code>
**untar !TraceToolkit stuff
**<code>tar -xzvf TraceToolkit-0.9.6pre3-plus.tar.gz</code>
*unpack and apply patch:
**<code>./tpm -f TraceToolkit-0.9.6pre3-plus.pl -o TraceToolkit</code>

Now, follow the build and usage instructions for the software in the include docs (!TraceToolkit/Help/index.html). For cross-compiling, use the instructions on this wiki page.

=== Utility programs ===

== How To Use ==
=== Building the software ===
Apply the ltt and relayfs patches to your kernel:

*configure kernel with LTT
*turn on RelayFS and LTT
make menuconfig
<code>File systems --->
Pseudo filesystems --->
*Relayfs file system support
(exit, exit)
General Setup --->
*Linux Trace Toolkit support
</code>
You can leave klog debugging support turned off.

Compile the user-space tracedaemon program:

I couldn't figure out if the tracevisualizer tools handle cross-compilation correctly (by which I mean that you can natively compile the tracevisualizer but cross-compile the tracedaemon). Instead of blindly trying configure tricks, I instead used the instructions from Karim's book "Building Embedded Linux Systems". In a nutshell, the instructions go something like this:

Overview:
* install source for user-space stuff
* configure for native build
* hand-build tracedaemon specifying a cross-compiler
* build rest of user-space suite using native compiler
* install programs as appropriate

Detail (in my case):

* install !TraceToolkit-0.9.6pre3, apply "plus" patch
** tpm -f user-space.pl -o !TraceToolkit-0.9.6pre3
* cd !TraceToolkit-0.9.6pre3
* configure --prefix=/home/tbird/work/ltt/tools
* make -C !LibUserTrace CC=<cross-compiler-gcc> !UserTrace.o
* make -C !LibUserTrace CC=<cross-compiler-gcc> LDFLAGS="-static"
* make -C Daemon CC=<cross-compiler-gcc> LDFLAGS="-static"
* cp Daemon/tracedaemon Daemon/Scripts/trace Daemon/Scripts/tracecore Daemon/Scripts/traceu <target file syste>/usr/sbin
* make
* make install

=== Using the software ===
In April, 2004, Karim wrote: [http://lkml.org/lkml/2004/4/3/100) lkml thread]
<code>
The documentation is out of date. Basically, the createdev.sh script isn't needed anymore because of relayfs. You need to mount relayfs to use LTT. See the classic dox on filesystem mounting for this kind of thing. It's
going to be something like: # mount -t relayfs nodev /mnt/relay

There's no insmod for LTT. It isn't a device driver module, following LKML recommendations.
</code>

== References ==
* LTT home page is at: [http://www.opersys.com/LTT/ LTT]
* Online documentation is at: [http://www.opersys.com/LTT/dox/ltt-online-help/index.html LTT Documentation]
** This documentation is a bit old (2002), and has parts that are out of date.

== Sample Results ==

== Future Work ==
Here is a list of things that could be worked on for this feature:
* online reference needs to be updated
** e.g. no mention of relayfs
* project seems fairly quiet, and unmaintained
* see if LTT patch should be refactored to take into account new kprobe support in the kernel

== LKML Reaction to tracing ==
In Sep 2002, there was a thread about tracing, where some major kernel developers expressed their concerns about tracing infrastructure in the kernel.

* [http://seclists.org/lists/linux-kernel/2002/Sep/5094.html Ingo Molnar]

<code>my problem with this stuff is conceptual: it introduces a constant drag on the kernel sourcecode, while 99% of development will not want to trace, ever. When i do need tracing occasionally, then i take those 30 minutes to write up a tracer from pre-existing tracing patches, tailored to specific problems.
...
so use the power of the GPL-ed kernel and keep your patches separate, releasing them for specific stable kernel ranches (or even development kernels).</code>

*[http://seclists.org/lists/linux-kernel/2002/Sep/5140.html Linus Torvalds]

<code>
> To summarize: You find tracing useful, but software tracing is only of limited value in areas you're working at. What about other developers, which only want to develop a simple driver, without having to understand the whole kernel? Traces still work where printk() or kgdb don't work. I think it's reasonable to ask an user to enable tracing and reproduce the problem, which you can't reproduce yourself.

That makes adding source bloat ok? I've debugged some drivers with dprintk() style tracing, and it often makes the code harder to follow, even if it ends up being compiled away.

From what I've seen from the LTT thing, it's too heavy-weight to be good for many things (taking SMP-global locks for trace events is _not_ a good idea if the trace is for doing things like doing performance tracing, where a tracer that adds synchronization fundamentally _changes_ what is going on in ways that have nothing to do with timing).

I suspect we'll want to have some form of event tracing eventually, but I'm personally pretty convinced that it needs to be a per-CPU thing, and the core mechanism would need to be very lightweight. It's easier to build up complexity on top of a lightweight interface than it is to make a lightweight interface out of a heavy one.
</code>

[[Category:Boot Time]
[[Category:Linux Trace Toolkit]]

← Older revision		Revision as of 00:39, 13 May 2009
Line 144:		Line 144:

	[[Category:Boot Time]]		[[Category:Boot Time]]
−	[[Category:~~Linux Trace Toolkit~~]]	+	[[Category:Tracing]]
		+	[[Category:Kernel]]

@@ Line 127: / Line 127: @@
 * [http://seclists.org/lists/linux-kernel/2002/Sep/5094.html Ingo Molnar]
-<code>my problem with this stuff is conceptual: it introduces a constant drag on the kernel sourcecode, while 99% of development will not want to trace, ever. When i do need tracing occasionally, then i take those 30 minutes to write up a tracer from pre-existing tracing patches, tailored to specific problems.
+<code>my problem with this stuff is conceptual: it introduces a constant drag on the kernel sourcecode, while 99% of development will not want to trace, ever. When i do need tracing occasionally, then i take those 30 minutes to write up a tracer from pre-existing tracing patches, tailored to specific problems.</source>
 ...
 so use the power of the GPL-ed kernel and keep your patches separate, releasing them for specific stable kernel  ranches (or even development kernels).</code>

@@ Line 143: / Line 143: @@
 </code>
-[[Category:Boot Time]
+[[Category:Boot Time]]
 [[Category:Linux Trace Toolkit]]

@@ Line 33: / Line 33: @@
 === Patch ===
 *Patches for Linux kernel version 2.6.8.1, and !TraceToolkit tarfile are here:
 **ltt-2.6.8.1.tar.gz

@@ Line 103: / Line 103: @@
 == References ==
 * LTT home page is at: [http://www.opersys.com/LTT/ LTT]
 * Online documentation is at: [http://www.opersys.com/LTT/dox/ltt-online-help/index.html LTT Documentation]
 ** This documentation is a bit old (2002), and has parts that are out of date.

@@ Line 116: / Line 116: @@
 == LKML Reaction to tracing ==
- In Sep 2002, there was a thread about tracing, where some major kernel developers expressed their concerns about tracing infrastructure in the kernel.
+In Sep 2002, there was a thread about tracing, where some major kernel developers expressed their concerns about tracing infrastructure in the kernel.
 * [http://seclists.org/lists/linux-kernel/2002/Sep/5094.html Ingo Molnar]

@@ Line 87: / Line 87: @@
   * make -C !LibUserTrace CC=<cross-compiler-gcc> LDFLAGS="-static"
   * make -C Daemon CC=<cross-compiler-gcc> LDFLAGS="-static"
-  * cp Daemon/tracedaemon Daemon/Scripts/trace Daemon/Scripts/tracecore Daemon/Scripts/traceu <target file syste>/usr/sbin
+  * cp Daemon/tracedaemon Daemon/Scripts/trace Daemon/Scripts/tracecore
   * make
   * make install